Quantization method and inference method for large language model, and electronic device

Abstract

The present application relates to the technical field of large language models, and discloses a quantization method and inference method for a large language model, and an electronic device. The quantization method for a large language model in embodiments of the present application comprises: for each linear layer to be quantized in a large language model, dividing channels of the linear layer in a hidden dimension into normal channels and outlier channels; performing, in a token dimension, INT8 quantization on a first activation matrix corresponding to the normal channels to obtain a second activation matrix, and performing, in an output channel dimension, INT4 quantization on a first weight matrix corresponding to the normal channels to obtain a second weight matrix; and determining an output result of the linear layer on the basis of the second activation matrix, the second weight matrix, a third activation matrix corresponding to the outlier channels, and a third weight matrix corresponding to the outlier channels.

Description

Quantization methods, inference methods, and electronic devices for large language models

[0001] Cross-references

[0002] This application claims priority to Chinese Patent Application No. 202510118310.1, filed on January 24, 2025, entitled "Quantization Method, Reasoning Method and Electronic Device for Large Language Models", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of large language model technology, and in particular relates to a quantization method, inference method and electronic device for large language models. Background Technology

[0004] Large Language Models (LLMs) have become the most popular language processing models due to their excellent performance in task processing. However, their large number of parameters also presents various challenges in practical application, such as high computational resource consumption and large memory footprint.

[0005] In response, related technologies have proposed model quantization schemes such as W8A8 quantization to reduce computational resource consumption and memory usage. However, taking the W8A8 quantization scheme as an example, although the model quantization accuracy is acceptable, the model still occupies a large amount of GPU memory. Therefore, how to reduce the GPU memory usage of large language models remains an urgent technical problem to be solved in this field. Summary of the Invention

[0006] The purpose of this application is to provide a method for quantizing large language models, an inference method, and an electronic device.

[0007] Firstly, a quantization method for a large language model is provided, comprising: for each linear layer to be quantized in the large language model, dividing the channels of the linear layer in the hidden layer dimension into normal channels and outlier channels; performing INT8 quantization on the first activation matrix corresponding to the normal channel in the token dimension to obtain a second activation matrix, and performing INT4 quantization on the first weight matrix corresponding to the normal channel according to the output channel to obtain a second weight matrix; determining the output result of the linear layer based on the second activation matrix, the second weight matrix, the third activation matrix corresponding to the outlier channel, and the third weight matrix corresponding to the outlier channel.

[0008] Secondly, a reasoning method for a large language model is provided. The large language model is obtained by quantization using the method described in the first aspect. The reasoning method includes: obtaining quantization configuration information corresponding to the large language model; loading and reasoning the model based on the quantization configuration information; wherein the quantization configuration information includes at least one of the following: first information, used to indicate the number of weight quantization bits corresponding to the linear layer in the large language model; second information, used to indicate whether the large language model is a floating-point model.

[0009] Thirdly, a quantization device for a large language model is provided, comprising: a first processing module, configured to divide the channels of each linear layer to be quantized in the large language model into normal channels and outlier channels in the hidden layer dimension; a second processing module, configured to perform INT8 quantization on the first activation matrix corresponding to the normal channel in the token dimension to obtain a second activation matrix, and to perform INT4 quantization on the first weight matrix corresponding to the normal channel according to the output channel to obtain a second weight matrix; and a third processing module, configured to determine the output result of the linear layer based on the second activation matrix, the second weight matrix, the third activation matrix corresponding to the outlier channel, and the third weight matrix corresponding to the outlier channel.

[0010] Fourthly, an inference apparatus for a large language model is provided, wherein the large language model is obtained by quantization using the method described in the first aspect, and the inference apparatus includes: a fourth processing module for acquiring quantization configuration information corresponding to the large language model; and a fifth processing module for loading and inferring the model based on the quantization configuration information; wherein the quantization configuration information includes at least one of the following: first information for indicating the number of quantization bits of the weights corresponding to the linear layers in the large language model; and second information for indicating whether the large language model is a floating-point model.

[0011] Fifthly, embodiments of this application provide an electronic device, including: a memory, a processor, and computer-executable instructions stored in the memory and executable on the processor, wherein the computer-executable instructions, when executed by the processor, implement the steps of the method described in the first aspect or the second aspect.

[0012] In a sixth aspect, embodiments of this application provide a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the steps of the method described in the first or second aspect. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 is one of the flowcharts of a large language model quantization method provided in an exemplary embodiment of this application.

[0015] Figure 2 is a schematic diagram of the quantization process provided in an exemplary embodiment of this application.

[0016] Figure 3a is a second schematic flowchart of a large language model quantization method provided in an exemplary embodiment of this application.

[0017] Figure 3b is a flowchart of the quantization method for a large language model provided in an exemplary embodiment of this application.

[0018] Figure 4 is a flowchart of a method for quantizing a large language model provided in an exemplary embodiment of this application.

[0019] Figure 5 is a flowchart of the quantization method for a large language model provided in an exemplary embodiment of this application.

[0020] Figure 6 is one of the flowcharts illustrating the reasoning method for a large language model provided in an exemplary embodiment of this application.

[0021] Figure 7a is one of the flowcharts illustrating a method for loading a quantized large language model provided in an exemplary embodiment of this application.

[0022] Figure 7b is a second schematic flowchart of a method for loading a quantized large language model provided in an exemplary embodiment of this application.

[0023] Figure 8 is a schematic diagram of the reasoning process of each linear layer in a large language model provided by an exemplary embodiment of this application.

[0024] Figure 9 is a schematic diagram of INT type conversion provided by an exemplary embodiment of this application.

[0025] Figure 10 is a second schematic flowchart of the reasoning method for a large language model provided in an exemplary embodiment of this application.

[0026] Figure 11 is a schematic diagram of the structure of a quantization device for a large language model provided in an exemplary embodiment of this application.

[0027] Figure 12 is a schematic diagram of the structure of a reasoning device for a large language model provided in an exemplary embodiment of this application.

[0028] Figure 13 is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0030] The quantization system and inference system mentioned below can be deployed on the same electronic device or on different electronic devices, without any restrictions.

[0031] Additionally, the “WxAy(Weight x-bit Activation y-bit)” mentioned below refers to the weight being quantized using INT x and the activation using INT y.

[0032] The technical solutions provided in this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0033] Figure 1 shows a flowchart of a large language model quantization method 100 provided in an embodiment of this application. This method 100 can be executed by a quantization system installed in an electronic device, wherein the electronic device can be a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. As shown in Figure 1, the method 100 may include the following steps.

[0034] Step S110: For each linear layer to be quantized in the large language model, the channels of the linear layer in the hidden dimensions are divided into normal channels and outlier channels.

[0035] The large language model mentioned in this embodiment can be understood as being obtained by stacking decoders, and most of the computation in the large language model is concentrated in the linear layer of the decoder. Therefore, the quantization of the large language model in this embodiment can be understood as quantizing the linear layer of the decoder.

[0036] Furthermore, for the large language model, considering that in some activated channels, regardless of token changes, outliers with large amplitude values ​​will continuously appear, these outliers determine the accuracy of the quantized large language model. Therefore, in this embodiment, during large language model quantization, the channels of the linear layer in the hidden layer dimension can be divided into normal channels and outlier channels based on outliers. Normal channels can be understood as channels corresponding to normal or ordinary points, while outlier channels can be understood as channels corresponding to outliers or anomalies. Finally, the corresponding weights are quantized according to the different normal and outlier channels to achieve the purpose of large language model quantization and ensure the accuracy of the quantized large language model.

[0037] In this embodiment, the normal point can be, but is not limited to, an activation value whose amplitude value (or absolute value) meets a specific condition. Correspondingly, the abnormal point can be, but is not limited to, an activation value whose amplitude value does not meet a specific condition. The specific condition may include, but is not limited to, an amplitude value less than a predetermined threshold.

[0038] In some embodiments, considering that for each linear layer, the channel where the outlier of the weight is located is determined by activation and corresponds to the position of the activated channel, in this embodiment, when dividing the channel of the linear layer in the hidden layer dimension into normal channels and outlier channels in step S110, the channel of the linear layer in the hidden layer dimension can be divided into normal channels and outlier channels according to activation statistics (such as the maximum value of the amplitude or absolute value of the activation).

[0039] That is, the process of dividing the channels of the linear layer in the hidden layer dimension into normal channels and outlier channels in step S110 may include: dividing the channels of the linear layer in the hidden layer dimension into normal channels and outlier channels according to the importance of each channel. The importance of a channel is determined based on the activation statistics corresponding to that channel. For example, the maximum absolute value of the activation of the large language model by channel can be obtained using a calibration dataset (such as PILE), where the larger the absolute value of the activation value, the higher the importance of its corresponding channel.

[0040] For example, referring to Figure 2, suppose that for the linear layer to be quantized, the calculation of the linear layer can be expressed as Y = XW T Its activation X∈R M×K Weights W∈R N×KM is the number of word segments on the input channel, K is the number of channels on the hidden layer dimension, and N is the number of output channels. When dividing the channels of the linear layer on the hidden layer dimension, the maximum value of the absolute value of each activation value X corresponding to each channel on the hidden layer dimension can be counted first. Then, the channels are sorted in ascending order according to the maximum value corresponding to each channel to obtain the sorting result, as shown in Figure 2 (2). Finally, a predetermined number of channels with lower sorting are selected as the outlier channels according to the sorting result, and the other channels besides the outlier channels are selected as the normal channels. Among them, by rearranging the activations by channel, the channels where the outlier points are located can be arranged at the end of the matrix. As shown in Figure 2, the part without background color represents the normal channel, that is, the channel corresponding to the weight and activation of the normal point, and the left diagonal shaded part represents the abnormal channel, that is, the weight and activation of the outlier point. The gray block in Figure 2 (3) illustrates the quantization granularity of the activation corresponding to the normal point and the quantization granularity of the weight corresponding to the normal point. In this embodiment, when performing channel reordering, the reorder index can be recorded in the linear layer for use in subsequent model inference, including activation dynamic reordering and quantization.

[0041] The predetermined quantity may be, but is not limited to, 256, etc.

[0042] Step S120: Perform INT8 quantization on the first activation matrix corresponding to the normal channel in the token dimension (per-token) to obtain the second activation matrix, and perform INT4 quantization on the first weight matrix corresponding to the normal channel in the output channel (per-channel) to obtain the second weight matrix.

[0043] In some embodiments, the quantization method for the third activation matrix and the third weight matrix corresponding to the outlier channel may include, but is not limited to, using no quantization or INT8 quantization for the third activation matrix and the third weight matrix to ensure the accuracy of the large language model after quantization.

[0044] If an unquantized approach is used, then the third activation matrix and the third weight matrix are floating-point matrices, such as floating-point data matrices in FP16 / BF16 format.

[0045] If INT8 quantization is used, the third activation matrix can be understood as a matrix after INT8 quantization in the token dimension, and the third weight matrix is ​​a matrix after INT8 quantization in the output channel dimension.

[0046] Step S130: Determine the output result of the linear layer based on the second activation matrix, the second weight matrix, the third activation matrix corresponding to the outlier channel, and the third weight matrix corresponding to the outlier channel.

[0047] In this embodiment, by using a mixed-precision quantization scheme based on W4A8 to quantize the activation and weights corresponding to the normal channels in the large language model, the GPU memory occupied by the large language model can be effectively reduced while ensuring quantization accuracy.

[0048] In some embodiments, the process of obtaining the second weight matrix by performing INT4 quantization on the first weight matrix corresponding to the normal channel according to the output channel in step S120 may include, but is not limited to: performing INT4 GPTQ (Generative Pretrained Transformer Quantization) quantization on the first weight matrix corresponding to the normal channel according to the output channel using the sorted Hessian matrix to obtain the second weight matrix.

[0049] In this embodiment, when performing INT4GPTQ quantization on the first weight matrix corresponding to the normal channel using a rearranged Hessian matrix for each output channel, the columns of the first weight matrix W are iterated. For example, for each column, all elements are quantized simultaneously, and after quantizing one column at a time, the other unquantized columns are adjusted. Since the same quantization order is used for each row in the first weight matrix W, by performing block quantization of the weights by column, the information of the Hessian inverse matrix stored in the Cholesky decomposition can be used to update the unquantized weights in the block, compensating for the loss caused by quantization. That is, in this embodiment, when quantizing the first weight matrix W based on GPTQ quantization, the second-order information of the Hessian matrix can be used to compensate for the introduced quantization error, thereby further improving the quantization accuracy of the first weight matrix corresponding to the normal channel.

[0050] Meanwhile, after the GPTQ process is completed on the first weight matrix corresponding to the normal channel, the introduced quantization error is further compensated to the weight part of the outlier point (the shaded part of the left diagonal line of the W matrix in (2) of Figure 2, which shows 3 columns), that is, the third weight matrix corresponding to the outlier channel. As mentioned above, in this embodiment, the third weight matrix corresponding to the outlier channel can be unquantized or quantized using INT8, which can ensure the quantization accuracy of the third weight matrix through higher precision representation, thereby improving the quantization accuracy of the language model.

[0051] In some embodiments, the process of determining the output result of the linear layer based on the second activation matrix, the second weight matrix, the third activation matrix corresponding to the outlier channel, and the third weight matrix corresponding to the outlier channel in step S130 may include, but is not limited to, steps S1301-S1303 shown in Figure 3a, as follows.

[0052] Step S1301: Determine the first intermediate matrix based on the second activation matrix and the second weight matrix, and perform inverse quantization on the first intermediate matrix to obtain a floating-point data matrix.

[0053] Optionally, the first intermediate matrix may be, but is not limited to, the result of multiplying the second activation matrix and the second weight matrix, as shown in shape (M, N) in Figure 2.

[0054] In this embodiment, considering that the second weight matrix is ​​obtained by INT4 quantization according to the output channel, while the second activation matrix is ​​obtained by INT8 quantization in the token dimension, that is, the data types of the second activation matrix and the second weight matrix are different, and matrix multiply and accumulate (MMA) can only be performed under the same data type, in this embodiment, before determining the first intermediate matrix based on MMA, the second weight matrix can be converted from INT4 type to INT8 type, and then the second activation matrix of type INT8 is multiplied with the second weight matrix to obtain the first intermediate matrix.

[0055] In some embodiments, when converting the second weight matrix from INT4 type to INT8 type, as shown in FIG3b, the INT4 type data in the second weight matrix is ​​logically shifted left by 4 bits to obtain INT8 type data, wherein the four 0s in the dashed line are obtained by left shifting.

[0056] Step S1302: Determine the second intermediate matrix based on the third activation matrix and the third weight matrix corresponding to the outlier channel.

[0057] In some embodiments, the second intermediate matrix may be, but is not limited to, the multiplication result obtained by multiplying the third activation matrix and the third weight matrix, as shown in shape(M, N) in Figure 2.

[0058] Step S1303: Determine the output result of the linear layer based on the floating-point data matrix and the second intermediate matrix.

[0059] In some embodiments, the method for determining the output result may differ depending on the type of the second intermediate matrix. For example, if the second intermediate matrix is ​​a floating-point data matrix, then the sum of the floating-point data matrix and the second intermediate matrix can be used as the output result; if the second intermediate matrix is ​​an INT8 type quantization matrix, then the second intermediate matrix can be dequantized into a floating-point data matrix first, and then the sum of the dequantized second intermediate matrix and the floating-point data matrix can be used as the output result.

[0060] In some embodiments, after the quantization of the large language model is completed through the foregoing embodiments, the large language model after the completion of the quantization of each of the linear layers, i.e., the floating-point model, and the quantization configuration information corresponding to the large language model can be directly saved for the inference system to load and infer the quantized large language model.

[0061] The quantization configuration information may include, but is not limited to, at least one of the first information and the second information. The first information is used to indicate the number of quantization bits for the weights corresponding to the linear layers in the large language model, such as the number of quantization bits {w_bit:4} corresponding to the second weight matrix.

[0062] The second information is used to indicate whether the large language model is a floating-point model, so that the inference system can determine the loading method of the large language model. For example, when the second information is {from_float:true}, the inference system can determine that the quantized large language model is a floating-point model, and therefore, the large language model can be loaded from the floating-point model. In this embodiment, the method of loading the large language model from the floating-point model is particularly useful for scenarios where only one quantization is needed to view the quantization effect, such as quantization exploration.

[0063] For example, when the second information is {from_float:false}, the inference system can determine that the quantized large language model is a non-floating-point model. Therefore, the large language model can be loaded from the non-floating-point model. Here, the non-floating-point model refers to the quantization system performing quantization compression on the quantized floating-point model.

[0064] In some embodiments, when the large language model indicated by the second information is a non-floating-point model, after the quantization system obtains a floating-point data matrix by dequantizing the first intermediate matrix in the aforementioned step S1301 (i.e., the quantized large language model is a floating-point model), it is necessary to further quantize and compress the floating-point model to obtain a non-floating-point model, which can also be called a compressed model.

[0065] In this embodiment, the process of further quantizing and compressing the floating-point model to obtain a non-floating-point model may include steps S140-S150 as shown in Figure 4.

[0066] Step S140: For each linear layer after quantization, perform INT4 round-to-nearest (RTN) symmetric quantization on the weight matrix corresponding to the normal channel in the linear layer according to the output channel to obtain the INT4 weight matrix, and package the INT4 weight matrix into a predetermined data format, and save the first quantization parameter corresponding to the INT4 weight matrix, such as FP16 / BP16.

[0067] The predetermined data format can be, but is not limited to, the UINT32 data format. In this embodiment, "packaging the INT4 weight matrix into a UINT32 data format" can be understood as having eight INT4 values ​​within a single UINT32 array.

[0068] Step S150: Perform INT8 RTN symmetric quantization on the weight matrix corresponding to the outlier channel in the linear layer according to the output channel to obtain the INT8 weight matrix, and package the INT8 weight matrix into the predetermined data format, and save the second quantization parameter corresponding to the INT8 weight matrix, such as FP16 / BP16.

[0069] The predetermined data format can be, but is not limited to, the UINT32 data format. In this embodiment, "packaging the INT4 weight matrix into a UINT32 data format" can be understood as having four INT8 values ​​within a single UINT32 array.

[0070] In this embodiment, through compression quantization in steps S140-S150, the inference system can directly load the compressed and quantized non-floating-point model when loading a large language model, thereby improving the model loading efficiency.

[0071] Based on the quantization method of the large language model provided in the aforementioned method embodiment 100, the quantization process of the large language model provided in this application embodiment will be illustrated below with reference to Figures 2 and 5.

[0072] Step S501: Obtain activation statistics.

[0073] (1) Load the large language model to be quantized into a computing unit, such as a central processing unit (CPU), and iteratively obtain all linear layers of the large language model. Register a hook function for the forward pass of each linear layer.

[0074] (2) Extract a subset from the calibration dataset as input to the large language model. For example, a subset of 512 samples can be extracted from the calibration dataset, with each sample having a maximum length of 512.

[0075] (3) Load the large language model into the graphics processing unit (GPU) layer by layer, and use the hook function to obtain the activation values ​​of each channel in the hidden layer dimension of each linear layer.

[0076] (4) Calculate the maximum value among the absolute values ​​of each activation value corresponding to each channel in the hidden layer dimension of the linear layer. For example, iterate through the absolute values ​​of each activation value in turn. If the maximum value of the absolute value of the next sample (i.e., activation) in this channel is greater than that of the previous sample, the larger value can be used as the statistical value of this channel. And so on, to obtain the activation statistics of each channel of the linear layer input under the subset, that is, the maximum value among the absolute values ​​of each activation value corresponding to each channel.

[0077] (5) For the linear layer, after completing the activation statistics of the linear layer, the linear layer is removed from the GPU and placed on the CPU, and (3)-(4) are repeated to obtain the activation statistics of each channel of each linear layer in the hidden layer dimension.

[0078] (6) Store the activation statistics in a dictionary (dict) structure, and the key value when storing is a unique name of each linear layer in the large language model.

[0079] Step S502: Sort the channels in ascending order according to the maximum value corresponding to each channel to obtain the sorting result, record the channel index of the reordering, such as reorder_index, and store the sorting result reorder_index in a dictionary with the name of the linear layer as the key value.

[0080] Step S503: Determine whether the sorting results of all linear layers in the large language model have been obtained, such as the reorder_index. If yes, proceed to step S504. Otherwise, for linear layers for which no sorting results have been obtained, proceed to step S502 again until the sorting results of all linear layers in the large language model have been obtained.

[0081] The large language model records the rearranged index of the channels (or activations) of each linear layer, which is used to rearrange the activations of each linear layer during model inference to ensure that they match the channel indexes corresponding to the weights.

[0082] Step S504: For each linear layer, select a predetermined number (e.g., 256) of the lower-ranked channels as the outlier channels according to the ranking result, and select the other channels besides the outlier channels as the normal channels.

[0083] Step S505: Determine whether the channel partitioning of each linear layer has been completed. If yes, proceed to step S506. Otherwise, for the linear layers that have not been partitioned, proceed to step S504 again until the channel partitioning on each linear layer is completed.

[0084] Step S506: For each linear layer, the first activation matrix corresponding to the normal channel is quantized by INT8 in the token dimension to obtain the second activation matrix, and the first weight matrix corresponding to the normal channel is quantized by INT4 GPTQ in the output channel using the sorted Hessian matrix to obtain the second weight matrix. The third weight matrix and the third activation matrix corresponding to the outlier channel are not quantized, or the third weight matrix and the third activation matrix corresponding to the outlier channel are quantized by INT8.

[0085] In this embodiment, all linear layers to be quantized can be understood as a block-structured decoding layer (DecoderLayer). The DecoderLayer can be loaded into the GPU for quantization.

[0086] Step S507: Determine whether the quantization of all linear layers has been completed. If yes, proceed to step S508. Otherwise, for the unquantized linear layers, proceed to step S506 again until the quantization of all linear layers is completed.

[0087] Step S508: Dequantize the INT type weights in each linear layer after quantization to floating-point type weights to obtain a floating-point type large language model. Save the large language model after quantization of each linear layer, i.e., the floating-point model, and save the quantization configuration information corresponding to the large language model. The second information in the quantization configuration information can be {from_float:true} to indicate that the large language model is a floating-point model.

[0088] Optionally, after obtaining the floating-point model, it can also be compressed and quantized through steps S140-S150 before being saved. Based on this, the second information in the quantization configuration information can be {from_float:false}, to indicate that the large language model is a compressed and quantized non-floating-point model.

[0089] The quantization scheme for the large language model provided in this embodiment loads the model into the GPU in blocks and rearranges the weights and activations according to activation statistics to obtain normal point parts (corresponding to normal channels) and outlier parts (corresponding to outlier channels). Then, dynamic INT8 quantization is performed on the first activation matrix corresponding to the normal channel, and error compensation is performed on the first weight matrix corresponding to the normal channel using the second-order information of the matrix layer by layer. Then, INT4 quantization is performed according to the output channel. The third activation matrix and third weight matrix corresponding to the outlier channel are quantized using either no quantization or INT8 quantization. This achieves the purpose of quantizing the large language model. At the same time, the aforementioned quantization process provided in this embodiment adopts a mixed precision quantization scheme based on W4A8, which can complete the model quantization process through a single GPU, reducing the memory usage of the quantization process. This allows the size of the quantized model to be compressed to 1 / 4 of the FP16 model, achieving the quantization compression effect of W4A16, while also restoring the precision of the quantized model and possessing INT8×INT8 inference computing capabilities.

[0090] Figure 6 illustrates a flowchart of a large language model inference method 600 provided in an embodiment of this application. This method 600 can be executed by an inference system installed in an electronic device, wherein the electronic device can be a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. As shown in Figure 6, the method 600 may include the following steps.

[0091] Step S610: Obtain the quantization configuration information corresponding to the large language model.

[0092] The large language model is a model obtained by quantization using the steps in the aforementioned method embodiment 100, and the large language model stores the rearranged index of the channels of each linear layer in the hidden layer dimension.

[0093] In some embodiments, the quantization configuration information corresponding to the large language model can be obtained based on the model address provided by the user.

[0094] Step S620: Load and infer the model based on the quantization configuration information.

[0095] The quantization configuration information may include, but is not limited to, at least one of the first information and the second information. The first information is used to indicate the number of quantization bits for the weights corresponding to the linear layers in the large language model, such as the number of quantization bits {w_bit:4} corresponding to the second weight matrix.

[0096] The second information is used to indicate whether the large language model is a floating-point model, so that the inference system can determine the loading method of the large language model. For example, when the second information is {from_float:true}, the inference system can determine that the quantized large language model is a floating-point model, and therefore, the large language model can be loaded from the floating-point model.

[0097] For example, when the second information is {from_float:false}, the inference system can determine that the quantized large language model is a non-floating-point model. Therefore, the large language model can be loaded from the non-floating-point model. Here, the non-floating-point model refers to the quantization system performing quantization compression on the quantized floating-point model.

[0098] In some embodiments, for the case of loading from a floating-point model, the process of loading the model according to the quantization configuration information in step S620 may include, but is not limited to, steps S6201-S6204 shown in FIG7a, as follows.

[0099] Step S6201: Determine that the second information (such as {from_float:true}) in the quantization configuration information indicates that the large language model is a floating-point model.

[0100] Step S6202: For each linear layer, the weight matrix corresponding to the normal channel in the linear layer is quantized by INT4 RTN according to the weight quantization bit number indicated by the first information to obtain an INT4 weight matrix. The INT4 weight matrix is ​​then packaged into a predetermined data format and the first quantization parameter corresponding to the INT4 weight matrix is ​​saved.

[0101] Step S6203: Perform INT8 RTN symmetric quantization on the weight matrix corresponding to the outlier channel in the linear layer according to the output channel to obtain the INT8 weight matrix, and package the quantized INT8 weight matrix into the predetermined data format, and save the second quantization parameter corresponding to the INT8 weight matrix.

[0102] In S6202-S6203, the outlier channels in the linear layer are a predetermined number of channels that are ranked low in the hidden layer dimension, and the normal channels are all channels in the hidden layer dimension of the linear layer other than the outlier channels. In this embodiment, the predetermined number is consistent with the predetermined number (e.g., 256) used by the aforementioned quantization system when quantizing large language models, thereby ensuring the consistency of the channel partitioning results between the inference system and the quantization system, and thus ensuring the accuracy of the inference results.

[0103] Step S6204: Load the large language model into a computing unit, such as a GPU, according to the INT4 weight matrix, the INT8 weight matrix, the first quantization parameter, and the second quantization parameter in the predetermined data format.

[0104] In this embodiment, steps S6201-S6204 involve a model loading scheme that directly loads the floating-point model. Specifically, the inference system first reads the floating-point model into the CPU's memory, quantizes the floating-point model, and automatically packages it into a compressed format (such as the aforementioned predetermined data format). Then, the compressed and quantized model is loaded into the GPU's video memory. This allows users to perform inference directly without worrying about the specific compression format of the large language model.

[0105] Furthermore, for the case of loading from a non-floating-point model, the process of loading the model according to the quantization configuration information in step S620 may include, but is not limited to, steps S6205-S6206 as shown in Figure 7b, as follows.

[0106] Step S6205: Determine that the second information in the quantization configuration information indicates that the large language model is a non-floating-point model.

[0107] Step S6206: Load the large language model into the computing unit according to the predetermined parameters corresponding to the large language model.

[0108] The predetermined parameters include an INT4 weight matrix, an INT8 weight matrix, a first quantization parameter corresponding to the INT4 weight matrix, and a second quantization parameter corresponding to the INT8 weight matrix under a predetermined data format.

[0109] In this embodiment, the steps of quantizing the floating-point model and automatically packaging it into a compressed format (such as the aforementioned predetermined data format) in the model loading method provided by S6205-S6206 have already been performed in the quantization system. That is, this embodiment loads directly from the non-floating-point model (from_float=false), or in other words, the inference system loads the compressed and quantized non-floating-point model. This loading method does not require weight quantization during model loading (or deployment), saving model deployment time. However, it requires users to have a deep understanding of the compression format of the quantized model in this system, or to use the compression and packaging tool provided by this system.

[0110] For the two model loading methods provided in steps S6201-S6204 and steps S6205-S6206 above, regardless of which model loading method is used, since the quantization scheme of the large language model in the aforementioned method embodiment 100 is adopted, when using the large language model for model loading and inference, its memory usage can be reduced to 1 / 4 of that of the FP16 / BF16 type model, while ensuring that the accuracy of the quantized model is almost undamaged.

[0111] In some embodiments, when performing inference based on a large language model after loading, the activation of each linear layer can be dynamically quantized in the token dimension according to the activation quantization method in the aforementioned quantization system. For example, the activation of the normal channel can be quantized in the token dimension using INT8, and the activation of the outlier channel can be quantized in the token dimension using INT8 or not quantized (consistent with the quantization method of the weights corresponding to the outlier channels).

[0112] In some embodiments, considering that activations need to be dynamically quantized during model inference, while weights are statically quantized by channel in the quantization system, in order to minimize inference latency, this embodiment can also integrate activation rearrangement, quantization, and other operations into existing or new operators of the large language model to improve the inference speed during model inference. The integration method is described below.

[0113] (1) A first operator is incorporated in the predecessor layer of at least one linear layer in the large language model, wherein the first operator is used to perform at least one of a rearrangement operation and a quantization operation on the activation of the at least one linear layer, wherein the rearrangement operation and / or quantization operation implemented by the first operator are static.

[0114] In some embodiments, the predecessor layer can be an existing layer or a newly added layer in the large language model. Of course, in this embodiment, when fusing rearrangement and quantization operations, efforts should be made to fuse the corresponding activated rearrangements in the current layer into the operators in the previous layer (predecessor layer), and new operators can be inserted where fusion is not possible. Specifically, if the rearrangement is fused into the operators of the predecessor layer, the rearrangement is static; if the rearrangement is only fused into the operations of the previous layer, the rearrangement is dynamic.

[0115] For example, assuming the first operator is used to rearrange the activations of the at least one linear layer, then, referring to Figure 8, if the activations of q_proj, k_proj, and v_proj are the same, the rearrangement of the activations of q_proj, k_proj, and v_proj can be fused into the LayerNorm layer before LlamaAttention to achieve dynamic rearrangement; if the rearrangement of the activation of o_proj cannot be fused into the previous layer, a reorder_and_requant operator can be added before o_proj to rearrange and quantize the activation of o_proj; if the activations of gate_proj and up_proj are the same, the rearrangement of the activations of gate_proj and up_proj can be fused into the LayerNorm after LlamaAttention; if the rearrangement of the activation of down_proj is fused into the linear layers gate_proj and up_proj before it, the output channels of the weights of gate_proj and up_proj are statically reordered.

[0116] For example, assuming the first operator is used to quantize the activations of the at least one linear layer, then, referring to Figure 8, the quantizations of the activations of q_proj, k_proj, and v_proj are fused in the LayerNorm layer before LlamaAttention; the quantization of the activation of o_proj cannot be integrated into the previous layer, so a reorder_and_requant operator is added before o_proj to quantize the activation of o_proj; the quantizations of the activations of gate_proj and up_proj are fused into the LayerNorm layer after LlamaAttention; and the quantization of the activation of down_proj is fused into the activation layer before it.

[0117] (2) At least one linear layer in the large language model incorporates a second operator, which is used to perform at least one of the following operations on the output of the linear layer: dequantization operation and INT type conversion operation.

[0118] For example, INT type conversion operations can be integrated into the linear layer operators corresponding to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.

[0119] Through the above fusion operations, the final inference operator functionality is shown in Figure 8. LayerNorm, in addition to performing layer normalization, also incorporates reordering and quantization (the corresponding operators are shown in gray). This operator requires recording the `reorder_index`. A `reorder_and_requant` operator needs to be inserted before `o_proj` specifically for reordering and quantization. The reorder index `reorder_index` can be recorded in its own LlamaAttention, because the `reorder_and_requant` operator only needs to exist in the inference system; its effect can be simulated when implementing the quantization algorithm. All linear layer operators, in addition to the MMA required for linear layers, also incorporate INT type conversion and dequantization operations (the corresponding operators are shown in right-side shading). The activation operator before `down_proj` incorporates quantization operations and is shown in left-side shading.

[0120] In some embodiments, taking the LLaMA model in a large language model as an example, please refer to Figure 8 again. Considering that the LLaMA model is composed of stacked LlamaDecoderLayer layers, in order to ensure the accuracy of the quantized large language model, during quantization, the activations of all linear layers in the LlamaDecoderLayer can be statistically sorted according to a specified index (such as the magnitude of the absolute value) using a calibration dataset to obtain the reorder_index. Then, during inference, the activations can be rearranged according to the reorder_index recorded in the large language model, as shown in Figure 2, placing outliers at the end of the matrix before calculation. The reordering of weights adopts static reordering, which is completed during model quantization, corresponding to the static reordering of hidden dimensions in Figure 8. For the reordering of activations, if they can be incorporated into the weights of the previous layer operator, static reordering is also used, corresponding to the static reordering of output channels in Figure 8; if they cannot be incorporated into the weights of the previous layer operator, only the reorder_index can be used to dynamically reorder the activations.

[0121] Based on this, after adopting the quantization and inference scheme provided in this application, the input of LlamaDecoderLayer is FP16 / BF16, and the input data of all linear layers in LlamaDecoderLayer is of type INT8, such as q_proj, k_proj, v_proj, and o_proj in the LlamaAteention structure, and gate_proj, up_proj, and down_proj in LlamaMLP. For each linear layer, the activations need to be quantized along the token dimension before the data is input to the linear layer. Considering that this application uses dynamic INT8 quantization for activations corresponding to normal channels in the linear layer, static INT4 quantization for weights corresponding to normal channels, and INT8 quantization (or no quantization) for activations and weights corresponding to outlier channels, the output of the linear layer is FP16 / BF16.

[0122] As shown in Figure 2, the normal channel and outlier channel are quantized separately. Furthermore, the input for matrix multiplication in the outlier channel is INT8, allowing for MMA, while the input for MMA in the normal channel is INT4 and INT8 respectively. Therefore, before MMA, INT4 needs to be converted to INT8, and then INT8 MMA can be performed. Finally, the MMA result is dequantized to FP16 / BF16.

[0123] In some embodiments, this embodiment also provides a conversion method for converting INT4 to INT8. As shown in Figure 9, considering that INT4 weights are signed INT4s packaged in UINT32 format data as required (i.e., eight INT4s in one UINT32), before performing MMA on the INT4 weights and INT8 activations corresponding to the normal channel of the linear layer, the compressed INT4 weights packaged in UINT32 need to be converted to INT8 format. As can be seen from the left side of Figure 9, converting INT4 to INT8 actually involves logically shifting the original INT4 number left by 4 bits; the four zeros in the dashed lines are obtained through this left shift.

[0124] For example, when converting INT4 weights in UINT32 to INT8 format, a fast INT type conversion operation can be performed as shown on the right side of 9. For instance, assuming that the UINT32 format stores 8 INT4 format weights, represented by W7, W6...W0 respectively, then a bitwise AND operation can be performed between the UINT32 format weights and 0xf0f0f0f0 to obtain the INT8 format corresponding to the odd-numbered INT4 format weights; or, a bitwise AND operation can be performed between the UINT32 format weights and 0x0f0f0f0f, followed by a logical left shift of 4 bits to obtain the INT8 format corresponding to the even-numbered INT4 format weights.

[0125] Based on the description of the aforementioned method embodiment 600, the reasoning process of the large language model provided in this application will be explained below with reference to Figure 10.

[0126] Step S1001: Obtain the quantized large language model and quantization configuration information based on the model address of the large language model provided by the user.

[0127] Step S1002: Determine whether the large language model is a floating-point model based on the second information in the quantization configuration information. If it is, first execute step S1003 to compress the quantized floating-point model, and then execute step S1004. Otherwise, directly execute step S1004.

[0128] Step S1003: Load the floating-point model into the CPU's memory, then load the weights of all linear layers in LlamaDecoderLayer into the GPU layer by layer, and quantize them layer by layer. Specifically, the weights corresponding to normal channels are symmetrically quantized using INT4 RTN according to the output channel. The quantized INT4 weights are packaged into UINT32 data format, and the first quantization parameter is saved. The weights corresponding to outlier channels (such as hidden layer dimensions of 256 channels) are symmetrically quantized using INT8 RTN according to the output channel. The quantized INT8 weights are packaged into UINT32 data format, and the second quantization parameter is saved.

[0129] Step S1004: Load the compressed quantized model (i.e., non-floating-point model) into the GPU's video memory.

[0130] The inference system employs symmetric per-channel quantization. Therefore, the parameters loaded by the inference system include: the INT4 weight matrix and its first quantization parameter for the normal channel, and the INT8 weight matrix and its second quantization parameter for the outlier channel. The INT4 weights in the INT4 weight matrix for the normal channel are stored in groups of eight in UINT32 data format, while the INT8 weights in the INT8 weight matrix for the outlier channel are stored in groups of four in UINT32 data format. The quantization parameter scales are FP16 / BF16.

[0131] In step S1005, the inference system performs inference based on the user's input using the large language model that has been loaded into the GPU memory.

[0132] Taking the LLaMA model in the large language model as an example, when using the large language model that has been loaded into the GPU memory for inference, the workflow during inference can be seen in Figure 8. For example, for each linear layer, the first activation matrix corresponding to the normal channel is dynamically quantized in the token dimension using INT8, and the third activation matrix corresponding to the abnormal channel is dynamically quantized in the token dimension using INT8 or not quantized. The quantization method can be referred to the relevant description in the aforementioned quantization system, which will not be repeated here.

[0133] In the inference scheme of the large language model provided in this embodiment, on the one hand, when using the quantized large language model, the user does not need to know the specific format of the model compression, but only needs to load the model according to the second information in the quantization configuration information; on the other hand, when performing model inference, for the activation and weight corresponding to the normal channel, the INT4 weight can be quickly converted to the INT8 weight during inference. Therefore, whether it is the activation and weight corresponding to the normal channel or the activation and weight corresponding to the outlier channel, the final calculation is performed using INT8×INT8 Tensor Core, which can improve the inference speed.

[0134] Figure 11 shows a schematic diagram of the quantization device for a large language model provided in an embodiment of this application. The device 1100 includes: a first processing module 1110, which divides the channels of each linear layer to be quantized in the large language model into normal channels and outlier channels in the hidden layer dimension; a second processing module 1120, which performs INT8 quantization on the first activation matrix corresponding to the normal channel in the token dimension to obtain a second activation matrix, and performs INT4 quantization on the first weight matrix corresponding to the normal channel according to the output channel to obtain a second weight matrix; and a third processing module 1130, which determines the output result of the linear layer based on the second activation matrix, the second weight matrix, the third activation matrix corresponding to the outlier channel, and the third weight matrix corresponding to the outlier channel.

[0135] In some embodiments, dividing the channels of the linear layer in the hidden layer dimension into normal channels and outlier channels includes: dividing the channels of the linear layer in the hidden layer dimension into normal channels and outlier channels according to the importance of each channel in the hidden layer dimension; wherein, the importance of the channel is determined according to the activation statistics corresponding to the channel.

[0136] In some embodiments, dividing the channels of the linear layer in the hidden layer dimension into normal channels and outlier channels based on the importance of each channel in the hidden layer dimension includes: calculating the maximum value among the absolute values ​​of each activation value corresponding to each channel in the hidden layer dimension; sorting the channels in ascending order according to the maximum value corresponding to each channel to obtain a sorting result; selecting a predetermined number of channels with lower sorting order as outlier channels according to the sorting result; and selecting the other channels besides the outlier channels as normal channels.

[0137] In some embodiments, the step of performing INT4 quantization on the first weight matrix corresponding to the normal channel to obtain the second weight matrix includes: performing INT4GPTQ quantization on the first weight matrix corresponding to the normal channel using a sorted Hessian matrix to obtain the second weight matrix.

[0138] In some embodiments, determining the output of the linear layer based on the second activation matrix, the second weight matrix, the third activation matrix corresponding to the outlier channel, and the third weight matrix corresponding to the outlier channel includes: determining a first intermediate matrix based on the second activation matrix and the second weight matrix; performing inverse quantization on the first intermediate matrix to obtain a floating-point data matrix; determining a second intermediate matrix based on the third activation matrix corresponding to the outlier channel and the third weight matrix; and determining the output of the linear layer based on the floating-point data matrix and the second intermediate matrix.

[0139] In some embodiments, determining the first intermediate matrix based on the second activation matrix and the second weight matrix includes: converting the second weight matrix from INT4 type to INT8 type; and multiplying the second activation matrix and the second weight matrix to obtain the first intermediate matrix.

[0140] In some embodiments, the floating-point data matrix is ​​a floating-point data matrix in FP16 / BF16 format.

[0141] In some embodiments, the third activation matrix and the third weight matrix are floating-point matrices; or, the third activation matrix is ​​a matrix after INT8 quantization in the token dimension, and the third weight matrix is ​​a matrix after INT8 quantization in the output channel dimension.

[0142] In some embodiments, after dequantizing the first intermediate matrix to obtain a floating-point data matrix, the third processing module 1130 is further configured to: for each of the linear layers after quantization, perform INT4 RTN symmetric quantization on the weight matrix corresponding to the normal channel in the linear layer according to the output channel to obtain an INT4 weight matrix, and package the INT4 weight matrix into a predetermined data format and save the first quantization parameter corresponding to the INT4 weight matrix; perform INT8 RTN symmetric quantization on the weight matrix corresponding to the outlier channel in the linear layer according to the output channel to obtain an INT8 weight matrix, and package the INT8 weight matrix into the predetermined data format and save the second quantization parameter corresponding to the INT8 weight matrix.

[0143] In some embodiments, the apparatus 1100 further includes: a model storage module, configured to store the large language model after quantization of each of the linear layers, and to store quantization configuration information corresponding to the large language model; wherein the quantization configuration information includes at least one of the following: first information, used to indicate the number of weight quantization bits corresponding to the linear layers in the large language model; second information, used to indicate whether the large language model is a floating-point model.

[0144] The device 1100 provided in this application embodiment can execute the methods described in the preceding method embodiment 100 and realize the functions and beneficial effects of the methods described in the preceding method embodiment 100, which will not be repeated here.

[0145] Figure 12 shows a schematic diagram of the structure of a quantization device 1200 for a large language model provided in an embodiment of this application. The device 1200 includes: a fourth processing module 1210, used to obtain quantization configuration information corresponding to the large language model; and a fifth processing module 1220, used to perform model loading and inference based on the quantization configuration information. The large language model is obtained by quantization using the methods described in method embodiment 100. The quantization configuration information includes at least one of the following: first information, used to indicate the number of weight quantization bits corresponding to the linear layer in the large language model; and second information, used to indicate whether the large language model is a floating-point model.

[0146] In some embodiments, loading the model according to the quantization configuration information includes: determining that the second information in the quantization configuration information indicates that the large language model is a floating-point model; for each linear layer, performing INT4 RTN symmetric quantization on the weight matrix corresponding to the normal channel in the linear layer according to the weight quantization bit number indicated by the first information to obtain an INT4 weight matrix, and packaging the INT4 weight matrix into a predetermined data format and saving the first quantization parameter corresponding to the INT4 weight matrix; performing INT8 RTN symmetric quantization on the weight matrix corresponding to the outlier channel in the linear layer according to the output channel to obtain an INT8 weight matrix, and packaging the quantized INT8 weight matrix into the predetermined data format and saving the second quantization parameter corresponding to the INT8 weight matrix; loading the large language model into the computing unit according to the INT4 weight matrix, the INT8 weight matrix, the first quantization parameter, and the second quantization parameter under the predetermined data format; wherein, the outlier channel in the linear layer is a predetermined number of channels that are ranked last in the hidden layer dimension of the linear layer, and the normal channel is the channel in the hidden layer dimension of the linear layer other than the outlier channel.

[0147] In some embodiments, loading the model according to the quantization configuration information includes: determining that the second information in the quantization configuration information indicates that the large language model is a non-floating-point model; loading the large language model into the computing unit according to predetermined parameters corresponding to the large language model; wherein the predetermined parameters include an INT4 weight matrix, an INT8 weight matrix, a first quantization parameter corresponding to the INT4 weight matrix, and a second quantization parameter corresponding to the INT8 weight matrix under a predetermined data format.

[0148] In some embodiments, a first operator is incorporated in the precursor layer of at least one linear layer in the large language model, wherein the first operator is used to perform at least one of a rearrangement operation and a quantization operation on the activation of the at least one linear layer.

[0149] In some embodiments, the precursor layer is an existing layer or a newly added layer in the large language model.

[0150] In some embodiments, at least one linear layer in the large language model incorporates a second operator, which is used to perform at least one of the following operations on the output of the linear layer: dequantization and INT type conversion.

[0151] The device 1200 provided in this application embodiment can execute the methods described in the preceding method embodiments and realize the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.

[0152] Figure 13 illustrates a schematic diagram of the hardware structure of the electronic device implementing the embodiments of this application. Referring to the figure, at the hardware level, the electronic device includes a processor and optionally, an internal bus, a network interface, and a memory. The memory may include RAM, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.

[0153] The processor, network interface, and memory can be interconnected via an internal bus, which can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus.

[0154] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0155] The processor reads the corresponding computer program from non-volatile memory into memory and then runs it, forming a device for locating a specific user at the logical level. The processor executes the program stored in memory and specifically performs the methods disclosed in the embodiments shown in Figures 1-6, achieving the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.

[0156] The methods disclosed in the embodiments shown in Figures 1-6 of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including GPUs, CPUs, network processors (NPs), etc.; it can also be a digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0157] The electronic device can also execute the methods described in the preceding method embodiments and achieve the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.

[0158] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0159] This application also proposes a computer-readable storage medium that stores one or more programs. When executed by an electronic device including multiple applications, the one or more programs cause the electronic device to perform the methods disclosed in the embodiments shown in Figures 1-6 and achieve the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0160] The computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc.

[0161] This application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, implement the following process: the method disclosed in the embodiments shown in Figures 1-6 and achieve the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.

[0162] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media do not include transient media, such as modulated data signals and carrier waves.

[0163] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0164] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0165] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A quantization method for large language models, comprising: For each linear layer to be quantized in the large language model, the channels of the linear layer in the hidden layer dimension are divided into normal channels and outlier channels; The first activation matrix corresponding to the normal channel is quantized by INT8 in the token dimension to obtain the second activation matrix, and the first weight matrix corresponding to the normal channel is quantized by INT4 according to the output channel to obtain the second weight matrix. The output of the linear layer is determined based on the second activation matrix, the second weight matrix, the third activation matrix corresponding to the outlier channel, and the third weight matrix corresponding to the outlier channel.

2. The method as described in claim 1, wherein, The step of dividing the channels of the linear layer in the hidden layer dimension into normal channels and outlier channels includes: Based on the importance of each channel in the hidden layer dimension of the linear layer, the channels in the hidden layer dimension of the linear layer are divided into normal channels and outlier channels; The importance of a channel is determined based on the activation statistics corresponding to that channel.

3. The method as described in claim 2, wherein, The step of dividing the channels of the linear layer in the hidden layer dimension into normal channels and outlier channels based on the importance of each channel in the hidden layer dimension includes: The maximum value among the absolute values ​​of each activation value corresponding to each channel in the hidden layer dimension of the linear layer is calculated. The channels are sorted in ascending order based on the maximum value corresponding to each channel to obtain the sorting result; Based on the sorting results, a predetermined number of channels that are ranked lower are selected as outlier channels, and the other channels besides the outlier channels are selected as normal channels.

4. The method of claim 1, wherein, The step of performing INT4 quantization on the first weight matrix corresponding to the normal channel to obtain the second weight matrix includes: The second weight matrix is ​​obtained by performing INT4 GPTQ quantization on the first weight matrix corresponding to the normal channel using the sorted Hessian matrix according to the output channel.

5. The method of claim 1, wherein, The step of determining the output of the linear layer based on the second activation matrix, the second weight matrix, the third activation matrix corresponding to the outlier channel, and the third weight matrix corresponding to the outlier channel includes: The first intermediate matrix is ​​determined based on the second activation matrix and the second weight matrix, and the first intermediate matrix is ​​dequantized to obtain a floating-point data matrix. The second intermediate matrix is ​​determined based on the third activation matrix and the third weight matrix corresponding to the outlier channel; The output of the linear layer is determined based on the floating-point data matrix and the second intermediate matrix.

6. The method of claim 5, wherein, The step of determining the first intermediate matrix based on the second activation matrix and the second weight matrix includes: Convert the second weight matrix from INT4 type to INT8 type; The first intermediate matrix is ​​obtained by multiplying the second activation matrix by the second weight matrix.

7. The method of claim 5, wherein, The floating-point data matrix is ​​a floating-point data matrix in FP16 / BF16 format.

8. The method of claim 5, wherein, The third activation matrix and the third weight matrix are floating-point matrices; Alternatively, the third activation matrix is ​​a matrix after INT8 quantization in the token dimension, and the third weight matrix is ​​a matrix after INT8 quantization in the output channel dimension.

9. The method of claim 5, wherein, After dequantizing the first intermediate matrix to obtain a floating-point data matrix, the method further includes: For each linear layer after quantization, the weight matrix corresponding to the normal channel in the linear layer is symmetrically quantized by the output channel using INT4 RTN to obtain an INT4 weight matrix. The INT4 weight matrix is ​​then packaged into a predetermined data format, and the first quantization parameter corresponding to the INT4 weight matrix is ​​saved. The weight matrix corresponding to the outlier channel in the linear layer is quantized by INT8 RTN according to the output channel to obtain the INT8 weight matrix, and the INT8 weight matrix is ​​packaged into the predetermined data format and the second quantization parameter corresponding to the INT8 weight matrix is ​​saved.

10. The method according to any one of claims 5-9, wherein, The method further includes: Save the large language model after quantization of each of the linear layers, and save the quantization configuration information corresponding to the large language model; The large language model records the rearranged index of the channels corresponding to each linear layer, and the quantization configuration information includes at least one of the following: The first piece of information is used to indicate the number of weight quantization bits corresponding to the linear layer in the large language model; The second piece of information is used to indicate whether the large language model is a floating-point model.

11. A reasoning method for a large language model, wherein the large language model is obtained by quantization using the method described in any one of claims 1 to 10, the reasoning method comprising: Obtain the quantization configuration information corresponding to the large language model; Model loading and inference are performed based on the quantized configuration information; The large language model records the rearranged index of the channels corresponding to each linear layer, and the quantization configuration information includes at least one of the following: The first piece of information is used to indicate the number of weight quantization bits corresponding to the linear layer in the large language model; The second piece of information is used to indicate whether the large language model is a floating-point model.

12. The method of claim 11, wherein, The step of loading the model according to the quantization configuration information includes: The second information in the quantization configuration information indicates that the large language model is a floating-point model; For each of the linear layers, the weight matrix corresponding to the normal channel in the linear layer is symmetrically quantized according to the output channel by the weight quantization bit number indicated by the first information to obtain the INT4 weight matrix, and the INT4 weight matrix is ​​packaged into a predetermined data format and the first quantization parameter corresponding to the INT4 weight matrix is ​​saved. The weight matrix corresponding to the outlier channel in the linear layer is symmetrically quantized by INT8 RTN according to the output channel to obtain the INT8 weight matrix, and the quantized INT8 weight matrix is ​​packaged into the predetermined data format and the second quantization parameter corresponding to the INT8 weight matrix is ​​saved. The large language model is loaded into the computing unit according to the INT4 weight matrix, the INT8 weight matrix, the first quantization parameter, and the second quantization parameter in the predetermined data format. Wherein, the outlier channels in the linear layer are a predetermined number of channels that are ranked later in the hidden layer dimension of the linear layer, and the normal channels are the channels in the hidden layer dimension of the linear layer other than the outlier channels.

13. The method of claim 11, wherein, The step of loading the model according to the quantization configuration information includes: The second information in the quantization configuration information indicates that the large language model is a non-floating-point model; The large language model is loaded into the computing unit according to the predetermined parameters corresponding to the large language model. The predetermined parameters include an INT4 weight matrix, an INT8 weight matrix, a first quantization parameter corresponding to the INT4 weight matrix, and a second quantization parameter corresponding to the INT8 weight matrix under a predetermined data format.

14. The method according to any one of claims 11-13, wherein, The preceding layer of at least one linear layer in the large language model incorporates a first operator, wherein the first operator is used to perform at least one of a rearrangement operation and a quantization operation on the activation of the at least one linear layer.

15. The method of claim 14, wherein, The precursor layer is either an existing layer or a newly added layer in the large language model.

16. The method according to any one of claims 11-13, wherein, At least one linear layer in the large language model incorporates a second operator, which is used to perform at least one of the following operations on the output of the linear layer: dequantization and INT type conversion.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as claimed in any one of claims 1-16.

18. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as claimed in any one of claims 1-16.

Images