Ai model quantization method and apparatus, and related device

Abstract

An AI model quantization method and apparatus, and a related device, relating to the technical field of artificial intelligence. The method comprises: determining a first smoothing scale, and using the first smoothing scale to update a parameter value in a first linear layer and a second linear layer, wherein the first smoothing scale is configured for reducing a quantization error generated by the parameter value in the first linear layer, at least one network layer is spaced between the first linear layer and the second linear layer, a computation order of the second linear layer during an inference process of an AI model precedes a computation order of the first linear layer, and output data of the second linear layer is transferred to the first linear layer by means of linear computation; and upon completion of a parameter value update for at least one linear layer in the AI model, quantizing the AI model. In this way, for two non-consecutive linear layers, a smoothing scale corresponding to a first linear layer is used to update a parameter value in the two linear layers without having to introduce additional smoothing operators, thereby ensuring quantized AI models can achieve a relatively high level of inference efficiency and inference precision.

Description

AI model quantization methods, devices and related equipment

[0001] This application claims priority to Russian patent application filed on January 9, 2025, with application number RU2025100054 and entitled "One Quantization Efficient solution based on iteration smoothing of outliers", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to an AI model quantification method, apparatus and related equipment. Background Technology

[0003] With the development of artificial intelligence (AI) technology, AI models such as large language models (LLMs) not only possess high inference performance but also include a large number of parameters, sometimes reaching hundreds of billions. In such cases, AI models can reduce data volume through compression techniques such as quantization. This typically involves quantizing the weights in the linear layers of the AI ​​model to reduce the memory space required during runtime, while also improving inference efficiency and reducing inference energy consumption. Quantization refers to converting high-precision parameters in the AI ​​model into low-precision parameters, such as converting weights from 32-bit floating-point numbers (high precision) to 8-bit integers (low precision).

[0004] In real-world applications, outliers may exist in the hidden states of an AI model. For example, during inference, the activation values ​​(tensor data) output by the hidden layers of an AI model might be data X, as shown in Figure 1, which is the input data to the linear layer. The values ​​in the second column are significantly larger than other activation values; for instance, activation value "16" is 16 times larger than activation value "1". Activation value "16" is an outlier. If the outlier is too large after quantizing the parameters in the AI ​​model, the result obtained by calculating the quantized parameter values ​​with the outlier may differ significantly from the result obtained by calculating the parameters with the outlier before quantization. This increases the quantization error of the AI ​​model, thereby reducing the inference accuracy of the quantized AI model.

[0005] Typically, smoothing operators, such as the Mul operator (used for element-wise multiplication), can be introduced before the linear layer to evenly distribute the quantization error between the weights of the linear layer and the hidden state (i.e., the input data of the linear layer). This process is called smooth quantization. However, while this smooth quantization method can guarantee the accuracy of the AI ​​model after quantization, the AI ​​model's inference efficiency is lower during the inference process because it needs to execute the additional introduced operators. Summary of the Invention

[0006] This application provides an AI model quantization method, aiming to improve the inference efficiency of the quantized AI model while ensuring its inference accuracy. Furthermore, this application also provides an AI model quantization device, a computing device, a computer-readable storage medium, and a computer program product.

[0007] Firstly, this application provides an AI model quantization method, which can be executed by a corresponding AI model quantization device. Specifically, the AI ​​model quantization device determines a first smoothing scale for a first linear layer in the AI ​​model. This first smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the first linear layer. The AI ​​model includes multiple consecutive network layers, including a first linear layer and a second linear layer. At least one network layer separates the first linear layer from the second linear layer. The second linear layer is computed before the first linear layer during the inference process of the AI ​​model, and its output data is passed to the first linear layer via linear computation. Then, the AI ​​model quantization device updates the parameter values ​​in the first and second linear layers using the first smoothing scale, and quantizes the AI ​​model after at least one linear layer in the AI ​​model has completed its parameter value update.

[0008] For the discontinuous first and second linear layers in the AI ​​model, the AI ​​model quantization device uses the smoothing scale corresponding to the first linear layer to update the parameter values ​​in both layers. This achieves the fusion of the smoothing operators of the first and second linear layers, eliminating the need to introduce an additional smoothing operator for the first linear layer in the AI ​​model. Consequently, this smoothing operator is not executed during AI model inference, thus improving the inference efficiency of the AI ​​model. Simultaneously, updating the parameter values ​​in the first linear layer using the smoothing scale reduces the quantization error generated after the parameter values ​​in the first linear layer are quantized, ensuring that the accuracy of the quantized AI model remains at a high level.

[0009] In one possible implementation, the multiple network layers in the AI ​​model also include a normalization layer, which is computed before the second linear layer during the AI ​​model inference process. Then, the AI ​​model quantization device can further determine a second smoothing scale for the second linear layer after updating the parameter values ​​using a first smoothing scale. This second smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the second linear layer. Furthermore, before quantizing the AI ​​model, the AI ​​model quantization device can also update the parameter values ​​in the second linear layer and the normalization layer using the second smoothing scale. Thus, by iteratively smoothing and quantizing multiple linear layers, the AI ​​model quantization device can avoid errors in the order of updating parameter values ​​in the linear layers, which could affect the inference accuracy of the quantized AI model.

[0010] In one possible implementation, the AI ​​model's multiple network layers also include a third linear layer, whose input data is the same as that of the second linear layer. Then, the AI ​​model quantization device can further determine a third smoothing scale for the third linear layer before quantizing the AI ​​model. This third smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the third linear layer. Specifically, when updating the parameter values ​​in the second linear layer and the normalization layer using the second smoothing scale, the AI ​​model quantization device can first calculate the average smoothing scale between the second and third smoothing scales, and then use this average smoothing scale to update the parameter values ​​in the normalization layer, the second linear layer, and the third linear layer. Thus, for multiple linear layers sharing the same input data, the AI ​​model quantization device calculates the average smoothing scale of each linear layer, thereby fusing the smoothing operators corresponding to the multiple linear layers with the normalization layer. This improves the accuracy of smoothing quantization for multiple linear layers, ensuring that the inference accuracy of the quantized AI model reaches a high level.

[0011] In one possible implementation, the AI ​​model's multiple network layers also include a fourth linear layer, which is computed before the normalization layer. Therefore, the AI ​​model quantization device can add a smoothing operator to the fourth linear layer, even if the normalization layer and the linear layer with parameter values ​​are not computed before it. This smoothing operator reduces the quantization error caused by the parameter values ​​in the fourth linear layer. Thus, for linear layers where smoothing operator fusion is not possible, the AI ​​model quantization device introduces an additional smoothing operator to ensure a high level of inference accuracy for the quantized AI model.

[0012] In one possible implementation, the AI ​​model includes an MLP (Multilayer Perceptron) block, which includes a normalization layer, a first linear layer, and a second linear layer, with the output of the normalization layer serving as the input of the second linear layer.

[0013] In one possible implementation, the AI ​​model includes a self-attention block, a first linear layer including a matrix multiplication operator for computing the output data of the self-attention block, and a second linear layer including a matrix multiplication operator for computing the value matrix in the input data of the self-attention block.

[0014] In one possible implementation, the first and second linear layers are based on a GQA (Group Query Attention) architecture. The input data of the first linear layer is obtained by concatenating the output data of multiple heads in the GQA architecture. The second linear layer includes matrix multiplication operators in multiple heads for calculating the value matrix. When the AI ​​model quantization device updates the parameter values ​​in the second linear layer using the first smoothing scale, it can specifically calculate a fourth smoothing scale based on the first smoothing scale, and update the parameter values ​​of the matrix multiplication operators based on the fourth smoothing scale. The value of each dimension in the fourth smoothing scale is calculated from the values ​​of multiple dimensions in the first smoothing scale. The number of dimensions of the first smoothing scale is the same as the number of dimensions of the input data of the first linear layer, and the number of dimensions of the fourth smoothing scale is the same as the number of dimensions of the parameters of the matrix multiplication operators. Thus, for the linear layers in the GQA architecture, the AI ​​model quantization device can integrate the smoothing scale of the first linear layer into the second linear layer by converting the high-dimensional smoothing scale into a low-dimensional smoothing scale, ensuring that the inference accuracy of the quantized AI model still reaches a high level.

[0015] In one possible implementation, when determining the first smoothing scale for the first linear layer in the AI ​​model, the AI ​​model quantization device may specifically utilize the AI ​​model to perform inference on a calibrated dataset. During the inference process, the input data for the first linear layer is determined, allowing the AI ​​model quantization device to calculate the first smoothing scale based on the input data and parameter values ​​of the first linear layer. Thus, based on the input data of the first linear layer generated during the actual inference process of the AI ​​model, a smoothing scale can be determined to reduce the quantization error of the parameter values ​​in the first linear layer. This allows the smoothing scale to be subsequently integrated into other linear layers, ensuring that the inference efficiency of the quantized AI model remains at a high level.

[0016] Secondly, this application provides an AI model quantization device, which includes: a determining module, used to determine a first smoothing scale for a first linear layer in the AI ​​model, the first smoothing scale being used to reduce the quantization error caused by the parameter values ​​in the first linear layer, the AI ​​model including a plurality of consecutive network layers, the plurality of network layers including a first linear layer and a second linear layer, with at least one network layer spaced between the first linear layer and the second linear layer, the computation order of the second linear layer in the inference process of the AI ​​model being prior to the computation order of the first linear layer, and the output data of the second linear layer being passed to the first linear layer through linear computation; an updating module, used to update the parameter values ​​in the first linear layer and the parameter values ​​in the second linear layer using the first smoothing scale; and a quantization module, used to quantize the AI ​​model after the parameter values ​​of at least one linear layer in the AI ​​model have been updated.

[0017] In one possible implementation, the multiple network layers further include a normalization layer, the normalization layer being computed before the second linear layer during AI model inference; the determination module is further configured to determine a second smoothing scale for the second linear layer after updating the parameter values ​​in the second linear layer using a first smoothing scale, the second smoothing scale being used to reduce the quantization error caused by the parameter values ​​in the second linear layer; the update module is further configured to update the parameter values ​​in the second linear layer and the parameter values ​​in the normalization layer using the second smoothing scale before quantizing the AI ​​model.

[0018] In one possible implementation, the multiple network layers further include a third linear layer, the input data of which is the same as that of the second linear layer; a determination module is further configured to determine a third smoothing scale for the third linear layer before quantizing the AI ​​model, the third smoothing scale being used to reduce the quantization error caused by the parameter values ​​in the third linear layer; and an update module is configured to: calculate an average smoothing scale between the second and third smoothing scales; and update the parameter values ​​in the normalization layer, the parameter values ​​in the second linear layer, and the parameter values ​​in the third linear layer using the average smoothing scale.

[0019] In one possible implementation, the multiple network layers further include a fourth linear layer, the computation order of which is prior to the computation order of the normalization layer; the AI ​​model quantization apparatus further includes an adding module for adding a smoothing operator to the fourth linear layer when the normalization layer and the linear layer with parameter values ​​are not included in the computation order of the fourth linear layer, the smoothing operator being used to reduce the quantization error caused by the parameter values ​​in the fourth linear layer.

[0020] In one possible implementation, the AI ​​model includes an MLP (Multilayer Perceptron) block, which includes a normalization layer, a first linear layer, and a second linear layer, with the output of the normalization layer serving as the input of the second linear layer.

[0021] In one possible implementation, the AI ​​model includes a self-attention block, a first linear layer including a matrix multiplication operator for computing the output data of the self-attention block, and a second linear layer including a matrix multiplication operator for computing the value matrix in the input data of the self-attention block.

[0022] In one possible implementation, the first linear layer and the second linear layer are based on a GQA (Group Query Attention) architecture. The input data of the first linear layer is obtained by concatenating the output data of multiple heads in the GQA architecture. The second linear layer includes matrix multiplication operators in multiple heads for calculating value matrices. The update module is used to: calculate a fourth smoothing scale according to a first smoothing scale, wherein the value of each dimension in the fourth smoothing scale is calculated from the values ​​of multiple dimensions in the first smoothing scale. The number of dimensions of the first smoothing scale is the same as the number of dimensions of the input data of the first linear layer, and the number of dimensions of the fourth smoothing scale is the same as the number of parameter dimensions of the matrix multiplication operator; and update the parameter values ​​of the matrix multiplication operator according to the fourth smoothing scale.

[0023] In one possible implementation, the determining module is configured to: use an AI model to infer on a calibrated dataset; determine the input data of a first linear layer during the inference process of the AI ​​model based on the dataset; and calculate a first smoothing scale based on the input data of the first linear layer and the parameter values ​​in the first linear layer.

[0024] The AI ​​model quantization device provided in the second aspect corresponds to the AI ​​model quantization method provided in the first aspect. Therefore, the technical effects of the second aspect and any implementation thereof can be found in the relevant descriptions of the technical effects of the first aspect and the corresponding implementation thereof, and will not be repeated here.

[0025] Thirdly, this application provides a computing device including a processor and a memory; wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to perform the operation steps of the AI ​​model quantization method described in the first aspect and any implementation thereof.

[0026] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operation steps of the AI ​​model quantization method described in the first aspect or any implementation thereof.

[0027] Fifthly, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operational steps of the AI ​​model quantization method described in the first aspect or any implementation thereof.

[0028] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0029] Figure 1 is a schematic diagram of the average distribution of quantization error between the input data (hidden state) and the weights;

[0030] Figure 2 is a schematic diagram of the structure of an exemplary computing cluster provided in this application;

[0031] Figure 3 is a schematic diagram of fusing the smoothing operator corresponding to the linear layer into the normalization layer adjacent to it;

[0032] Figure 4a is a schematic diagram of fusing the smoothing operator corresponding to linear layer c to a non-adjacent linear layer b in an MLP block;

[0033] Figure 4b is a schematic diagram of fusing the smoothing operator corresponding to the linear layer d to the non-adjacent linear layer c in the self-attention block;

[0034] Figure 5 is a flowchart illustrating an AI model quantization method provided in this application;

[0035] Figure 6a is a schematic diagram of fusing the smoothing operators corresponding to linear layer a and linear layer b into the normalization layer in the MLP block;

[0036] Figure 6b is a schematic diagram of fusing the smoothing operators corresponding to linear layers a, b and c into the normalization layer in the self-attention block.

[0037] Figure 7 shows the test results of the Baichuan 2-13B model before and after quantization, including inference accuracy and acceleration effect.

[0038] Figure 8a is a schematic diagram of the smoothing operator fusion based on three iterations using multiple linear layers with the SDXL architecture;

[0039] Figure 8b is a schematic diagram of the CLIP index before and after iterative smoothing quantization of the AI ​​model of the SDXL architecture.

[0040] Figure 9a is a schematic diagram of the shared value tensor and key tensor of multiple attention heads within the same group in the GQA architecture;

[0041] Figure 9b is a schematic diagram of copying the value tensor and key tensor of some attention heads to other attention heads;

[0042] Figure 9c shows the linear layer final The smoothing scale is used to calculate the linear layer. value A schematic diagram of the smooth scale;

[0043] Figure 9d is a schematic diagram of the test results of the inference accuracy of the LLaMa3-8B model and the ChatGLM2-6B-32K model using the GQA architecture before and after quantization.

[0044] Figure 10 is a schematic diagram of the structure of an AI model quantization device provided in this application;

[0045] Figure 11 is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0046] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, various non-limiting embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained based on the embodiments in this application and based on the above content are within the scope of protection of this application.

[0047] Referring to Figure 2, a schematic diagram of a cluster structure is shown. As shown in Figure 2, cluster 20 may include multiple computing nodes and a quantization device 200. Figure 2 illustrates an example of cluster 20 including four computing nodes (computing node 101 and computing node 104, respectively).

[0048] In this cluster, the computing nodes refer to nodes with data computing capabilities, which can be implemented using processors or computing devices including processors. Furthermore, different computing nodes can communicate with each other through switching nodes, such as electrical switches or optical switches.

[0049] For example, a processor can be any type of processor or any combination thereof, such as a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a data processing unit (DPU). An accelerator, for example, can be a graphics processing unit (GPU), a neural network processing unit (NPU), or a tensor processing unit (TPU).

[0050] Cluster 20 can deploy AI models, such as different network layers of the AI ​​model deployed on different computing nodes. Furthermore, multiple computing nodes in cluster 20 can support the operation of the AI ​​model to provide corresponding inference services, such as text recognition services and dialogue services. For example, the AI ​​model can be a large language model (LLM), a large language model meta AI (LLaMA), a bidirectional encoder representations from transformers (BERT), or a generative pre-trained Transformer 3 (GPT-3) model, or other types of models such as GPT-4, etc., without limitation.

[0051] In practical applications, AI models typically have large parameter sizes, leading to higher resource consumption for cluster 20 to support their runtime, such as increased memory usage. Therefore, before deploying the AI ​​model to cluster 20, the quantization device 200 can quantize the parameter values. For example, it can convert weight values ​​represented by 32-bit floating-point numbers (high precision) to 8-bit integers (low precision), reducing the memory (and computational) resources required for storing and calculating these parameter values. For instance, the quantized parameter values ​​can be those from linear layers in the AI ​​model; parameter values ​​from nonlinear layers may not be quantized (quantization could significantly impact the model's accuracy). Alternatively, both linear and nonlinear layer parameter values ​​can be quantized; there is no limitation on this.

[0052] However, during the inference process of the AI ​​model, there may be outliers in the input data of the linear layer. Furthermore, if the outlier is too large, the result obtained by calculating the quantized parameter value with the outlier may differ significantly from the result obtained by calculating the parameter value before quantization with the outlier. This will increase the quantization error of the AI ​​model, thereby reducing the inference accuracy of the quantized AI model.

[0053] In practical applications, the quantization device 200 can introduce a smoothing operator, such as the Mul operator (used for element-wise multiplication), before the linear layer. This smoothing operator can reduce outliers in the input data and amplify the parameter values ​​calculated with the outliers in the linear layer. This achieves an even distribution of quantization error between the parameter values ​​of the linear layer and the input data (or hidden states) of the linear layer, thereby mitigating the reduction in inference accuracy of the quantized AI model.

[0054] For example, suppose the parameter value in the linear layer is the weight matrix W shown above in Figure 1, the input of the linear layer is the input data X shown above in Figure 1, and the linear calculation logic in the linear layer is matrix multiplication of the input data X and the weight matrix W, that is, the input data of the linear layer Y = XW. Each data point (1≤j≤4) in the j-th column of the input data X will be multiplied sequentially with the data in the j-th row of the weight matrix W. Then, the quantization device 200 can add the smoothing operator corresponding to the following formula (1) before the linear layer to calculate the reduction ratio S for the j-th column of the input data X. seale_j The S scale_j That is, the scaling ratio of the j-th row of data in the weight matrix W, and using this S scale_jThe j-th column of the input data X is reduced in size, while the j-th row of the weight matrix W is increased in size. scale_j =max(|X j |) α / max(|W j |) 1-α Formula (1)

[0055] Where α is a hyperparameter. When α When the value of is 0.5, the above formula (1) becomes the following formula (2).

[0056] At this time, S scale_j It is the square root of the quotient of the maximum absolute value in the j-th column of the input data X and the maximum absolute value in the j-th row of the weight matrix W.

[0057] After the quantization device 200 performs smoothing using the above formula (2), the smoothed input data X and the weight matrix W are shown in Figure 1 below.

[0058] However, adding a smoothing operator before the linear layer increases the number of operators the AI ​​model needs to execute during inference, which increases inference latency and affects inference efficiency. Currently, the quantization device 200 can reduce the number of operators the AI ​​model needs to execute during inference by fusing the smoothing operator for the current linear layer to the adjacent preceding linear layer or normalization layer.

[0059] In practical implementation, assuming the AI ​​model includes adjacent linear layers and normalization layers (i.e., no other network layers exist between the linear and normalization layers), as shown in Figure 3, the quantization device 200 can provide a calibration dataset (such as a representative dataset calibrated by technicians) to the AI ​​model and use the AI ​​model for inference based on this calibration dataset. During the AI ​​model inference process, the quantization device 200 can determine the output of the normalization layer, i.e., the input data of the linear layer, and calculate the scaling ratio for each column of the output based on the output and the current parameter values ​​of the linear layer, obtaining a scaling matrix S, where the values ​​of each element are known. Therefore, the quantization device 200 can integrate the scaling matrix S into the operator of the normalization layer (the data output by the normalization layer is scaled based on the scaling matrix S each time), and update the parameter values ​​in the linear layer according to the scaling matrix S, such as updating the weight matrix in the linear layer above Figure 1 to the weight matrix shown below Figure 1. In this way, the AI ​​model can avoid executing additional smoothing operators during the inference process, thus ensuring that the inference efficiency of the AI ​​model can still reach a high level.

[0060] However, in real-world applications, AI models may contain multiple linear layers, and some linear layers may not have an adjacent preceding normalization layer or any other adjacent preceding linear layers, as shown in Figures 4a and 4b. In such cases, if linear layer A is adjacent to a normalization layer or linear layer B, the quantization device 200 will fuse the additional smoothing operator introduced for linear layer A into the normalization layer or linear layer B based on the aforementioned method. For this portion of linear layers (without any adjacent preceding linear layers), the quantization device 200 may introduce an additional smoothing operator for this portion of linear layers without fusing it with any other preceding network layers. This results in lower inference efficiency for the quantized AI model. Conversely, if the quantization device 200 does not introduce an additional smoothing operator for this portion of linear layers, the inference accuracy of the AI ​​model will decrease due to the quantization of the parameter values ​​in this portion of linear layers.

[0061] Based on this, in the cluster 20 provided in this application, the quantization device 200 integrates the smoothing operator introduced for some linear layers with the prior linear layers, thereby improving the inference efficiency of the quantized AI model while ensuring that the inference efficiency of the AI ​​model can still reach a high level.

[0062] In specific implementation, the quantization device 200 determines a smoothing scale for the first linear layer in the AI ​​model. This smoothing scale can be the scaling matrix mentioned above, used to reduce the quantization error caused by the parameter values ​​in the first linear layer. The AI ​​model includes multiple network layers, with the first linear layer being one of these layers. The multiple network layers also include a second linear layer that is computed earlier in the computation order. The computation order of the linear layers refers to their order of computation during the AI ​​model's inference process. Furthermore, there is at least one network layer between the first and second linear layers, as shown in Figure 4a, including linear layer c (first linear layer), linear layer b (second linear layer), etc., with a Mul layer (including operators for element-wise multiplication) separating linear layer b and linear layer c. Furthermore, the output data of the second linear layer can be passed to the first linear layer through linear computation. Therefore, scaling the output data of the second linear layer can achieve a proportional scaling of the input data of the first linear layer. The quantization device 200 can then update the parameter values ​​of the second linear layer based on this linear computation transfer relationship, using the smoothing scale determined for the first linear layer, to integrate the smoothing scale (i.e., the smoothing operator) into the second linear layer. Additionally, the quantization device 200 also updates the parameter values ​​of the first linear layer using the smoothing scale determined for the first linear layer, thereby using the parameter values ​​in the first linear layer to allocate part of the quantization error. Thus, after at least one linear layer in the AI ​​model has completed its parameter value update, the quantization device 200 quantizes the AI ​​model, specifically converting the high-precision parameter values ​​in the AI ​​model into low-precision parameter values.

[0063] For the discontinuous first and second linear layers (i.e., the two linear layers are not adjacent) in the AI ​​model, the quantization device 200 uses the smoothing scale corresponding to the first linear layer to update the parameter values ​​in both layers. This enables the fusion of the smoothing operator of the first linear layer and the smoothing operator of the second linear layer, eliminating the need to introduce an additional smoothing operator for the first linear layer in the AI ​​model. Therefore, the quantized AI model does not need to execute this smoothing operator during inference, thereby improving the inference efficiency of the quantized AI model. Simultaneously, updating the parameter values ​​in the first linear layer using the smoothing scale of the first linear layer reduces the quantization error generated after the parameter values ​​in the first linear layer are quantized, allowing the accuracy of the quantized AI model to still reach a high level.

[0064] For example, the quantization device 200 described above can be implemented in software or hardware. When implemented in software, the quantization device 200 can be program code running in a computing instance, such as a process or application. When implemented in hardware, the quantization device 200 can be a processor or a computing device including a processor. This application does not limit the specific implementation of the quantization device 200 in this regard.

[0065] It is important to note that the cluster 20 shown in Figure 2 above is merely an illustrative example and is not intended to limit the scope. For instance, other possible computing clusters may include more or fewer computing nodes. Alternatively, cluster 20 may include other types of nodes, such as management nodes or scheduling nodes. Management nodes can be used to manage the computing power of the computing nodes in cluster 20, such as putting some computing nodes in cluster 20 into hibernation to reduce power consumption. Scheduling nodes can be used to schedule computing power for AI models, i.e., to schedule which computing nodes in cluster 20 to run the AI ​​model; different AI models can be scheduled to different computing nodes in cluster 20.

[0066] For ease of understanding, the embodiments for quantizing AI models provided in this application are described below with reference to the accompanying drawings.

[0067] Referring to Figure 5, which is a flowchart illustrating an exemplary AI model quantization method provided in an embodiment of this application, the AI ​​model quantization method shown in Figure 5 can be applied to cluster 20 shown in Figure 2, or to other possible clusters. For ease of understanding and description, the following explanation uses cluster 20 shown in Figure 2 as an example.

[0068] As shown in Figure 5, the AI ​​model quantization method can specifically include the following steps.

[0069] S501: Quantization device 200 determines the first smoothing scale for the first linear layer in the AI ​​model.

[0070] The first smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the first linear layer. That is, after updating the parameter values ​​in the first linear layer using the first smoothing scale, the parameter values ​​in the first linear layer are quantized. Compared with directly quantizing the parameter values ​​in the first linear layer, the quantization error is relatively small.

[0071] The AI ​​model to be quantized by the quantization device 200 may include multiple consecutive network layers, including multiple linear layers. For ease of description, this embodiment uses a first linear layer and a second linear layer as examples. The first linear layer and the second linear layer are separated by at least one network layer, such as a Mul layer or a Reshape layer (used to change the shape of data without changing its content). Furthermore, the computation order of the second linear layer in the AI ​​model's inference process is before that of the first linear layer; that is, during the AI ​​model's inference process, the second linear layer is executed first, followed by the first linear layer. The output data of the second linear layer can be passed to the first linear layer through linear computation. At this time, the input data of the first linear layer is obtained by processing the output data of the second linear layer. Therefore, scaling operations on the input data of the first linear layer can be passed to the output data of the second linear layer through the logic of this linear computation.

[0072] As an example of determining the smoothing scale, before quantizing the AI ​​model, the AI ​​model can be deployed to multiple computing nodes in cluster 20 and run using these nodes. Then, the quantization device 200 can input a calibration dataset into the AI ​​model, which may include at least one data point, so that the AI ​​model can perform inference based on the calibration dataset. During AI model inference, the quantization device 200 can acquire the input data of the first linear layer and calculate the first smoothing scale of the first linear layer based on the input data and the parameter values ​​included in the first linear layer (used to perform corresponding calculations based on the input data). For example, the quantization device 200 can calculate the first smoothing scale based on the above formula (1) or formula (2).

[0073] In practical applications, the quantization device 200 can also determine the first smoothing scale in other ways. For example, it can calculate multiple different smoothing scales for the first linear layer based on different calibration datasets, and calculate the average of these multiple smoothing scales (tensor data) to obtain the first smoothing scale. In this embodiment, the specific implementation method of the quantization device 200 in determining the first smoothing scale is not limited.

[0074] S502: The quantization device 200 updates the parameter values ​​in the first linear layer and the parameter values ​​in the second linear layer using a first smoothing scale.

[0075] In this embodiment, after the quantization device 200 calculates the first smoothing scale for the first linear layer, it can use the first smoothing scale to update the parameter values ​​in the first linear layer.

[0076] For example, the quantization device 200 can perform a dot product operation between the first smoothing scale and the parameter values ​​in the first linear layer, and use the calculated result as the updated value of the parameter in the first linear layer.

[0077] For example, suppose the quantization device 200 calculates the smoothing scale S of the parameter values ​​in the j-th row of the weight matrix for the first linear layer using the formula (1) above. scale_j Then, the quantization device 200 can update the parameter values ​​in the first linear layer based on formula (3).

[0078] Among them, W 1_j The values ​​of the parameters in the j-th row of the weight matrix of the first linear layer before they are updated; S represents the updated values ​​of the parameters in the j-th row of the weight matrix of the first linear layer; scale_j This is the smoothing scale for the parameter values ​​in the j-th row of the weight matrix of the first linear layer.

[0079] Furthermore, the quantization device 200 can also use the first smoothing scale to update the parameter values ​​in the second linear layer, thereby integrating the smoothing operator for the first linear layer into the second linear layer.

[0080] In a specific implementation, the quantization device 200 can perform a dot product operation between the first smoothing scale and the parameter values ​​in the second linear layer, and use the calculated result as the updated value of the parameter in the second linear layer.

[0081] For example, suppose the quantization device 200 calculates the smoothing scale S of the parameter values ​​in the j-th row of the weight matrix for the first linear layer using the formula (1) above. scale_j This is the smoothing scale for the j-th column of the input data in the first linear layer. Therefore, the quantization device 200 can update the parameter values ​​in the second linear layer based on formula (4).

[0082] Among them, W 2_j The values ​​of the j-th column parameters in the weight matrix of the second linear layer before they are updated; The updated value of the parameter in the j-th column of the weight matrix of the second linear layer.

[0083] It is understandable that although there is at least one network layer between the first linear layer and the second linear layer, there is linear computational logic between them, allowing the output data of the second linear layer to be transmitted to the first linear layer through linear computation. Therefore, the quantization device 200 reduces the parameter values ​​in the weight matrix of the second linear layer, enabling the output data of the second linear layer to be reduced proportionally, and further reducing the input data of the first linear layer proportionally, thereby integrating the smoothing operator introduced for the first linear layer into the second linear layer.

[0084] To facilitate understanding, the following section will use two specific structures in AI models to illustrate how to fuse a smoothing operator from one linear layer into another.

[0085] Example 1: An AI model includes one or more multilayer perceptron (MLP) blocks. Each MLP block's subgraph can be shown on the left side of Figure 4a, including a layer normalization layer, linear layer a (including matrix multiplication operators for gating computation), linear layer b (including matrix multiplication operators), a Swish layer (including activation functions for nonlinear transformation), a multiplication layer (including the Mul operator), and a linear layer c (including matrix multiplication operators). Linear layer b is the second linear layer mentioned above, and linear layer c is the first linear layer mentioned above. The matrix multiplication operators in linear layers a, b, and c all include weight matrices (parameters). The multiplication layer multiplies the output data of the Swish layer with the output data of linear layer b, and the result is used as the input data of linear layer c. The output data of the normalization layer is used as the input data of linear layers a and b.

[0086] Then, the quantization device 200 can fuse the smoothing operator introduced for linear layer c (as shown in the middle of Figure 4a) into linear layer b. In this process, the quantization device 200 can first determine the first smoothing scale for linear layer c according to the above formula (1), update the parameter values ​​in linear layer c according to formula (3), and update the parameter values ​​in linear layer b according to formula (4), thereby realizing the fusion of the smoothing operator corresponding to linear layer c into linear layer b. The subgraph of the fused MLP block is shown on the right side of Figure 4a.

[0087] Example 2: The AI ​​model includes one or more self-attention blocks. Each self-attention block's subgraph can be as shown on the left side of Figure 4b, including linear layers a, b, c, and d, four reshape layers, a transpose layer (used to perform transpose operations on the data), two multiplication layers (including the MatMul operator), and a softmax layer. Linear layers a, b, c, and d all include weight matrices. The multiplication layers (excluding the weight matrices) are used to perform corresponding multiplication operations on the two sets of input data. Furthermore, linear layer a is used to calculate the query matrix, linear layer b is used to calculate the key matrix, linear layer c is used to calculate the value matrix, linear layer d is the first linear layer mentioned above, and linear layer c is the second linear layer mentioned above.

[0088] Then, the quantization device 200 can fuse the smoothing operator introduced for the linear layer d (as shown in the middle of Figure 4b) into the linear layer c. In this process, the quantization device 200 can first determine the first smoothing scale for the linear layer d according to the above formula (1), update the parameter values ​​in the linear layer d according to formula (3), and update the parameter values ​​in the linear layer c according to formula (4), thereby realizing the fusion of the smoothing operator corresponding to the linear layer d into the linear layer c. The subgraph of the fused self-attention block is shown on the right side of Figure 4b.

[0089] It is understood that steps S501 and S502 described above mainly introduce the process of fusing the smoothing operator corresponding to the first linear layer to the non-adjacent second linear layer. In practical application scenarios, the AI ​​model may include multiple linear layers, and for other linear layers, the quantization device 200 can refer to the above method to fuse the smoothing operators corresponding to other linear layers to the earlier (ordered according to the calculation logic) linear layer (which may be adjacent or non-adjacent). For details, please refer to the relevant descriptions above, which will not be repeated here.

[0090] Furthermore, some linear layers in the AI ​​model may not have a prior linear layer with which smoothing operators can be fused. For example, in the MLP block shown in Figure 4a above, linear layer b does not have a prior linear layer with which smoothing operators can be fused. In this case, the quantization device 200 can fuse the smoothing operator introduced for this part of the linear layer b into the prior normalization layer. For example, it can fuse the smoothing operator corresponding to linear layer b into the normalization layer (Layernorm). The following describes the implementation process of fusing the smoothing operator corresponding to the second linear layer into the normalization layer with reference to the accompanying drawings. Referring to Figure 5, this embodiment may further include the following steps.

[0091] S503: After updating the parameter values ​​in the second linear layer using the first smoothing scale, the quantization device 200 determines a second smoothing scale for the second linear layer, which is used to reduce the quantization error caused by the parameter values ​​in the second linear layer.

[0092] The specific implementation of determining the second smoothing scale in step S503 is similar to the specific implementation of determining the first smoothing scale in step S501 above. Please refer to the relevant part of the description in step S501 above, and it will not be repeated here.

[0093] S504: Before quantizing the AI ​​model, the quantization device 200 also updates the parameter values ​​in the second linear layer and the parameter values ​​in the normalization layer using the second smoothing scale.

[0094] In the AI ​​model, the normalization layer is computed before the second linear layer in the AI ​​model's inference process. For example, as shown in Figure 4a, the output data of the normalization layer can be used as the input data of the linear layer b. Thus, in the AI ​​model's inference process, the normalization layer will perform the data computation process before the linear layer b.

[0095] For example, when updating the parameter values ​​in the second linear layer, the quantization device 200 can perform a dot product operation between the second smoothing scale and the parameter values ​​in the second linear layer, and use the calculated result as the updated value of the parameter in the second linear layer.

[0096] For example, suppose the quantization device 200 calculates the smoothing scale of the parameter values ​​in the j-th row of the weight matrix for the second linear layer using the above formula (1). (i.e., the second smoothing scale mentioned above), then the quantization device 200 can update the parameter values ​​in the second linear layer again based on formula (5).

[0097] in, For the parameters in the j-th row of the weight matrix of the second linear layer, the smoothing scale is... The value before the update; For the parameters in the j-th row of the weight matrix of the second linear layer, the smoothing scale is... The updated value.

[0098] Furthermore, the quantization device 200 can also use the second smoothing scale to update the parameter values ​​in the normalization layer, thereby integrating the smoothing operator for the first linear layer into the second linear layer.

[0099] In practice, the quantization device 200 can perform a dot product operation between the second smoothing scale and the parameter values ​​in the normalization layer, and use the calculated result as the updated value of the parameter in the normalization layer.

[0100] For example, quantization device 200 can update the parameter values ​​in the normalization layer based on formula (6).

[0101] Among them, S ln_j The values ​​of the j-th column parameters in the weight matrix of the normalized layer before they are updated; The updated values ​​of the parameters in the j-th column of the weight matrix of the normalized layer.

[0102] It should be noted that, since smoothing quantization of linear layers that are later in the computation order will affect the parameter values ​​of linear layers that are earlier in the computation order, in practical applications, the quantization device 200 can iteratively update the parameter values ​​of multiple linear layers in the AI ​​model during the smoothing quantization process. This avoids the reduction in the inference accuracy of the quantized AI model due to an error in the update order of the parameter values ​​in the linear layers.

[0103] For example, regarding the normalization layer, the second linear layer, and the first linear layer in the aforementioned AI model, if the quantization device 200 first determines a second smoothing scale for the second linear layer and uses this second smoothing scale to update the parameter values ​​in the second linear layer and the normalization layer, and then the quantization device 200 determines a first smoothing scale for the first linear layer and uses this first smoothing scale to update the parameter values ​​in the first linear layer and the second linear layer, then the quantization device 200 may generate a large quantization error when quantizing the first linear layer or the second linear layer based on the updated parameter values, resulting in lower inference accuracy of the quantized AI model.

[0104] Therefore, in this embodiment, the quantization device 200 can perform smooth quantization on the linear layers with later calculation order first, and then on the linear layers with earlier calculation order, according to the calculation logic (including calculation order) between multiple linear layers, through multiple iterations. Specifically, the quantization device 200 first performs a first iteration to perform smooth quantization on the first linear layer with later calculation order, so as to update the parameter values ​​in the first linear layer and the parameter values ​​in the second linear layer. Then, the quantization device 200 performs a second iteration to perform smooth quantization on the second linear layer after parameter update, so as to update the parameter values ​​in the second linear layer again, and update the parameter values ​​in the normalization layer, thereby achieving correct updating of parameter values ​​in multiple linear layers.

[0105] As an example of implementation, the quantization device 200 can determine whether each of the multiple linear layers has a prior linear layer that can be fused to a smooth scale, based on the computational logic (such as network layer subgraphs) between the multiple linear layers.

[0106] If present, the quantization device 200 can perform smooth quantization on each linear layer sequentially in the reverse direction of the calculation order among the multiple linear layers. After completing the smooth quantization of these linear layers, for the remaining linear layers, if there is no preceding linear layer with a fused smooth scale before the calculation order of the linear layer, but there is a normalization layer with a fused smooth scale, the quantization device 200 can perform smooth quantization on the linear layer in the manner described above, specifically by updating the parameter values ​​in the linear layer and the parameter values ​​in the normalization layer.

[0107] If there are no preceding linear layers or normalization layers with fused smooth scales before the computation sequence of this linear layer, this linear layer is referred to as the fourth linear layer. The quantization device 200 can then add a smoothing operator to this fourth linear layer, such as the Mul operator. This smoothing operator is used to reduce the quantization error caused by the parameter values ​​in the fourth linear layer, ensuring that the inference accuracy of the quantized AI model can still reach a high level (in practical applications, the number of such linear layers is relatively small; therefore, adding a smoothing operator to such linear layers by the quantization device 200 will have virtually no impact on the inference efficiency of the AI ​​model).

[0108] In other implementations, the quantization device 200 may also be able to eliminate the need to add a smoothing operator to the linear layer, and there is no limitation on this.

[0109] In a further possible implementation, multiple linear layers in the AI ​​model may share the same input data. For example, in the MLP architecture shown in Figure 4a, the input data of linear layer a and the input data of linear layer b are both output data of the normalized layer; or, in a cross-attention architecture, the same data may be input to linear layers in multiple different cross-attention blocks. In this case, the quantization device 200 can combine the smoothing scales of these multiple linear layers to adjust the parameter values ​​in each linear layer, as well as the parameter values ​​in the linear layers or normalized layers preceding the computation order of these multiple linear layers (the computation order in the inference process of the AI ​​model). For example, the quantization device 200 can fuse the smoothing scales determined separately for multiple cross-attention blocks into the same linear layer or normalized layer located preceding the computation order of these multiple cross-attention blocks.

[0110] Taking an AI model comprising a second linear layer and a third linear layer, where the input data for both the second and third linear layers is the output data of the same normalization layer, as an example, before quantizing the AI ​​model, the quantization device 200 can determine not only a second smoothing scale for the second linear layer but also a third smoothing scale for the third linear layer. This third smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the third linear layer. The specific implementation of the quantization device 200 determining the third smoothing scale can be found in the description related to determining the first smoothing scale above, and will not be repeated here.

[0111] Then, the quantization device 200 can calculate the average smoothing scale between the second smoothing scale and the third smoothing scale. For example, the quantization device 200 can calculate the average value (where i is a positive integer) between the value of the i-th column in the second smoothing scale and the value of the i-th column in the third smoothing scale, and use this average value as the value of the i-th column in the average smoothing scale to calculate the average smoothing scale.

[0112] Finally, the quantization device 200 can use the average smoothing scale to update the parameter values ​​in the normalization layer, the second linear layer, and the third linear layer respectively. For the specific implementation, please refer to the description of the relevant parts of updating the parameter values ​​in the normalization layer and the second linear layer using the second smoothing scale, which will not be repeated here.

[0113] For example, for the MLP block shown on the right side of Figure 4a, the quantization device 200 can first perform a first iteration on linear layer c, calculate the smoothing scale corresponding to linear layer c, and use the smoothing scale to update the parameter values ​​in linear layer c and linear layer b, thereby fusing the smoothing operator of linear layer c into linear layer b, as shown on the left side of Figure 6a. Then, the quantization device 200 performs a second iteration on linear layers a and b, calculates the smoothing scale of linear layer a and the smoothing scale of linear layer b, and further calculates the average smoothing scale of these two smoothing scales, and uses the average smoothing scale to update the parameter values ​​in the normalization layer, thereby fusing the smoothing operators corresponding to linear layers a and b into the normalization layer, as shown on the right side of Figure 6a.

[0114] For example, for the attention block shown on the right side of Figure 4b, the quantization device 200 can first perform a first iteration on linear layer d, calculate the smoothing scale corresponding to linear layer d, and use the smoothing scale to update the parameter values ​​in linear layer d and linear layer c, thereby fusing the smoothing operator of linear layer d into linear layer c, as shown on the left side of Figure 6b. Then, the quantization device 200 performs a second iteration on linear layers a, b, and c, respectively, calculates the smoothing scale of linear layers a, b, and c, and further calculates the average smoothing scale of these three smoothing scales, and uses the average smoothing scale to update the parameter values ​​in the normalization layer, thereby fusing the smoothing operators corresponding to linear layers a and b into the normalization layer, as shown on the right side of Figure 6b.

[0115] S505: After the parameter values ​​of multiple linear layers in the AI ​​model have been updated, the quantization device 200 quantizes the AI ​​model.

[0116] In practical applications, when the parameter values ​​in the normalization layer of the AI ​​model are also updated, the quantization device 200 can specifically perform quantization on the AI ​​model after the parameter values ​​of multiple linear layers and the corresponding normalization layers in the AI ​​model have been updated.

[0117] For example, the quantization device 200 can specifically quantize the parameter values ​​in multiple linear layers of the AI ​​model from high-precision values ​​to low-precision values, thereby effectively reducing the resources consumed by the AI ​​model during inference and achieving effective compression of the AI ​​model.

[0118] In some possible implementations, the quantization device 200 may quantize the AI ​​model using gradient-based post-training quantization (GPTQ) or quantization to INT4 with GPU kernel support (QUIK) algorithms, or may use other quantization algorithms to quantize the AI ​​model, and there is no limitation on this.

[0119] Thus, based on the above method, the quantization device 200 integrates the smoothing operator introduced for the linear layer into other linear layers or normalization layers before the computation sequence. This allows the quantized AI model to avoid executing the smoothing operator during inference, thereby improving the inference efficiency of the AI ​​model. Simultaneously, integrating the smoothing operator corresponding to the linear layer into other linear layers or normalization layers reduces the quantization error caused by the quantization of parameter values ​​in that linear layer, ensuring that the accuracy of the quantized AI model remains at a high level. Furthermore, quantizing the AI ​​model converts the parameter values ​​in the AI ​​model from a high-precision representation to a low-precision representation, enabling effective compression of the AI ​​model.

[0120] In actual testing scenarios, for the Baichuan 2-13B model (an AI model whose parameter values ​​are represented by 32-bit floating-point numbers), after quantizing the Baichuan 2-13B model using methods 1, 2, and the method shown in Figure 5 above (i.e., method 3), the Baichuan 2-13B model before quantization, the Baichuan 2-13B model obtained by quantization based on method 1, the Baichuan 2-13B model obtained by quantization based on method 2, and the Baichuan 2-13B model obtained by quantization based on method 3 can be tested using the C-Eval dataset, and the test results are shown in Figure 7.

[0121] In this method, the Baichuan 2-13B model is quantized using method 1. That is, when a part of the linear layer does not have an adjacent preceding linear layer or a preceding normalization layer, no additional smoothing operator is introduced for that linear layer (nor is the process of fusing the smoothing operator into other network layers performed). The parameter values ​​in that part of the linear layer are directly quantized into low-precision parameter values, which results in a large quantization error in the parameter values ​​of that part of the linear layer.

[0122] Method 2 is used to quantize the Baichuan 2-13B model. That is, when some linear layers do not have adjacent preceding linear layers or preceding normalization layers, an additional smoothing operator is introduced for these linear layers, but the process of fusing the smoothing operator into other network layers is not performed. As a result, the smoothing operator introduced for these linear layers will generate a large computational delay.

[0123] As shown in Figure 7, after quantizing the Baichuan 2-13B model based on the method shown in Figure 5 (i.e., method 3), the accuracy of the Baichuan 2-13B model did not decrease (and even slightly improved), and the inference efficiency was significantly improved.

[0124] It should be noted that the embodiment shown in Figure 5 above is only an implementation example and is not intended to limit the scope. Other implementation examples based on the embodiment shown in Figure 5 are described below.

[0125] Example 1: In other embodiments, before quantizing the AI ​​model, the quantization device 200 may only fuse the smoothing operators corresponding to some linear layers into other non-adjacent linear layers, or may not perform the fusion of the smoothing operators corresponding to the linear layers into the normalization layer.

[0126] Example 2, in the embodiment shown in Figure 5, only describes the process of updating the parameter values ​​in the network layer through two iterations. In other embodiments, the network structure in the AI ​​model may include a larger number of linear layers. In this case, the quantization device 200 can update the parameter values ​​through three or more iterations to fuse the smoothing operators corresponding to multiple linear layers into the prior network layer.

[0127] For example, assuming the AI ​​model uses a stable diffusion XL (SDXL) architecture, the structure between some network layers in the AI ​​model is shown in Figure 8a. Between the MLP block and the ResNet layer, there is a linear layer d, an addition layer (used to perform addition on two inputs) without parameter values, and a reshape layer. The network layers included in the MLP block can be shown in Figure 4a. The ResNet layer refers to the residual network layer, which can solve the gradient vanishing and gradient exploding problems during the training of deep networks, improving the training efficiency and performance of the model. The ResNet layer includes operators for normalization calculations.

[0128] Then, the quantization device 200 can perform the first iteration according to the computational logic between network layers, first determining a smoothing scale 1 for linear layer d, and updating the parameter values ​​in linear layer d and linear layer c in the MLP block according to the smoothing scale 1. Then, the quantization device 200 performs the second iteration, determining a smoothing scale 2 for linear layer c after parameter value updates, and updating the parameter values ​​in linear layer c and linear layer b in the MLP block according to the smoothing scale 2. Finally, the quantization device 200 performs the third iteration, determining a smoothing scale 3 for linear layer b after parameter value updates, determining a smoothing scale 4 for linear layer a in the MLP block, and calculating the average smoothing scale between smoothing scale 3 and smoothing scale 4, thereby updating the parameter values ​​in linear layers a and b, as well as the parameter values ​​in the normalized layer, according to the average smoothing scale.

[0129] In actual testing scenarios, the SDXL architecture AI model was iteratively smoothed and quantized based on the above process. Different numbers of samples from the COCO-2014 and Parti datasets were used to run the SDXL architecture AI models before and after quantization, yielding the test results shown in Figure 8b. The contrastive language-image pre-training (CLIP) metric of the quantized AI model was similar to that of the unquantized model, meaning that the inference accuracy of the AI ​​model did not decrease due to quantization (in fact, it slightly improved). The CLIP metric is an indicator used to evaluate the degree to which the generated image matches the text description; generally, a higher CLIP value indicates higher inference accuracy of the AI ​​model.

[0130] Example 3: In practical applications, AI models may include a grouped-query attention (GQA) architecture. As shown in Figure 9a, in a GQA architecture, multiple attention heads are divided into multiple groups, with each group containing more than one attention head. For example, a GQA architecture could include 32 attention heads, which would be divided into two groups, each containing 16 attention heads. Each attention head would store its own query tensor, and multiple attention heads within each group would share the same value tensor and key tensor.

[0131] In the GQA architecture, the linear layer used to compute value tensors value With linear layers used to compute bond tensors key The required value and key tensors for some attention heads can be computed. The computed value and key tensors can be copied multiple times and distributed to the remaining attention heads, ensuring each attention head receives a copy of both. As shown in Figure 9b, assuming the GQA architecture includes 32 attention heads, designated h1 to h2, this is a separate set of attention heads. 32 Then, linear layer value With linear layer key We can calculate value tensor 1 and key tensor 1 based on some parameter values ​​of attention head h1, and then calculate attention head h based on the remaining parameter values. 17 Calculate value tensor 2 and key tensor 2. Then, the value tensor 1 and key tensor 1 calculated for attention head h1 can be copied 15 times and assigned to attention heads h2 through h1. 16 ; for attention head h 17 The calculated value tensor 2 and key tensor 2 can be copied 15 times and assigned to the attention head h. 18 Attention h32 Thus, attention head h1 to attention head h 16 It can perform calculations based on the same value tensor 1 and key tensor 1, but different query tensors; attention head h 17 Attention h 32 It will also perform calculations based on the same value tensor 2 and key tensor 2, but with different query tensors.

[0132] Additionally, the GQA architecture can also include a contact layer and a linear layer. final The connection layer is used to concatenate the outputs of multiple attention heads (e.g., concatenating 32-dimensional tensor data from 32 attention heads into a 1024-dimensional tensor); the linear layer... final Used to perform linear calculations on the output data of the connection layer (such as matrix multiplication of the output data using the weight matrix).

[0133] This makes the linear layer value (and linear layers) key Dimensions of parameter values ​​in ) ， It will be smaller than the linear layer final The dimension of the parameter values, thus the quantization device 200 utilizes a linear layer. final Corresponding smooth scale update linear layer value When setting parameter values, there may be a problem of parameter dimension mismatch.

[0134] Based on this, in a further possible implementation, the quantization device 200 is used for linear layers. final Calculate the smoothing scale final Then, based on this smoothing scale final Calculate the smoothing scale value Among them, smoothing scale value The values ​​of each dimension are scaled smoothly. final The smoothing scale is obtained through numerical calculations of multiple dimensions. final The number of dimensions and linear layers final The input data has the same number of dimensions, and the smoothing scale is... value The number of dimensions and linear layers value The number of parameter dimensions of the matrix multiplication operator in the model is the same. Then, the quantization device 200 is based on the smoothing scale. value Update the linear layer value The parameter values ​​of the matrix multiplication operator in the linear layer. final The quantization device 200 can be adjusted according to the smoothing scale. final Update linear layer final The parameter values ​​in the code. This allows for the implementation of linear layers within the GQA architecture. final The corresponding smoothing operators are fused into the linear layer.value middle.

[0135] Taking the GQA architecture with 32 attention heads as an example, these 32 attention heads are divided into two groups, and the output data of each attention head has a dimension of 32. Therefore, the linear layer... value The parameter values ​​have 64 dimensions (i.e., 32*2), where 32 dimensions are used for the attention heads h1 to h2 within the first group. 16 The computation tensor is 1, and the parameter values ​​of the remaining 32 dimensions are used for the attention head h in the second group. 17 Attention h 32 Computational tensor 2. Linear layer final The dimension of the intermediate parameter value is 1024 (i.e., 32*32). The quantization device 200 calculates the smoothing scale in the 1024-dimensional dimension. final Then, based on this smoothing scale final Calculate the smoothing scale in 64 dimensions. value As shown in Figure 9c, smoothing scale value The parameter value of the first dimension in the model can be the smoothing scale. final Attention head h1 to attention head h 16 The average of the parameter values ​​(16 parameter values) corresponding to the output data of the first dimension of each attention head is the smoothing scale. final The average value of the parameter values ​​in the 1st, 33rd, 65th, ..., 993rd dimensions; smoothing scale. value The parameter value of the 17th dimension can be the smoothing scale. final Attention head h 17 Attention h 32 The average of the parameter values ​​(16 parameter values) corresponding to the output data of the first dimension of each attention head is the smoothing scale. final The average value of the parameter values ​​in the 17th, 49th, 81st, ..., 1009th dimensions. Smoothing scale. value The calculation method for the parameter values ​​of the other dimensions is the same.

[0136] In actual testing scenarios, the LLaMa3-8B and ChatGLM2-6B-32K models, which employ the GQA architecture, were tested using the BoolQ and C-Eval datasets, respectively. The test results are shown in Figure 9d. As shown in Figure 9d, after iterative smoothing quantization of the LLaMa3-8B and ChatGLM2-6B-32K models based on the above method, the inference accuracy of the models is no lower than that before quantization (and may even be slightly improved).

[0137] In practical applications, the above-mentioned implementation of iterative smooth quantization for AI models can achieve a high level of inference accuracy even when the parameter values ​​in the AI ​​model are quantized to 8 bits or even 4 bits of data precision, and can also achieve a significant compression of the AI ​​model.

[0138] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0139] The AI ​​model quantization method provided in the embodiments of this application has been described above with reference to Figures 1 to 9d. Next, the structure of the AI ​​model quantization device and computing device provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0140] Referring to Figure 10, a schematic diagram of an AI model quantization device is shown. The AI ​​model quantization device 1000 includes:

[0141] The determination module 1001 is used to determine a first smoothing scale for the first linear layer in the AI ​​model. The first smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the first linear layer. The AI ​​model includes multiple consecutive network layers, including a first linear layer and a second linear layer. There is at least one network layer between the first linear layer and the second linear layer. The calculation order of the second linear layer in the inference process of the AI ​​model is before the calculation order of the first linear layer. The output data of the second linear layer is passed to the first linear layer through linear calculation.

[0142] Update module 1002 is used to update the parameter values ​​in the first linear layer and the parameter values ​​in the second linear layer using the first smoothing scale;

[0143] The quantization module 1003 is used to quantize the AI ​​model after the parameter values ​​have been updated in at least one linear layer in the AI ​​model.

[0144] In one possible implementation, the multiple network layers also include a normalization layer, which is computed before the second linear layer during the AI ​​model inference process.

[0145] The determination module 1001 is further configured to determine a second smoothing scale for the second linear layer after updating the parameter values ​​in the second linear layer using the first smoothing scale. The second smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the second linear layer.

[0146] The update module 1002 is also used to update the parameter values ​​in the second linear layer and the parameter values ​​in the normalization layer using a second smoothing scale before quantizing the AI ​​model.

[0147] In one possible implementation, the multiple network layers further include a third linear layer, the input data of which is the same as the input data of the second linear layer;

[0148] The determination module 1001 is also used to determine a third smoothing scale for the third linear layer before quantizing the AI ​​model. The third smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the third linear layer.

[0149] Update module 1002, used for:

[0150] Calculate the average smoothing scale between the second and third smoothing scales;

[0151] The parameter values ​​in the normalized layer, the second linear layer, and the third linear layer are updated using the average smoothing scale.

[0152] In one possible implementation, the multiple network layers further include a fourth linear layer, the computation order of which is prior to the computation order of the normalization layer;

[0153] The AI ​​model quantization device 1000 also includes an addition module 1004, which adds a smoothing operator to the fourth linear layer when the normalization layer and the linear layer with parameter values ​​are not included before the calculation order of the fourth linear layer. The smoothing operator is used to reduce the quantization error caused by the parameter values ​​in the fourth linear layer.

[0154] In one possible implementation, the AI ​​model includes an MLP (Multilayer Perceptron) block, which includes a normalization layer, a first linear layer, and a second linear layer, with the output of the normalization layer serving as the input of the second linear layer.

[0155] In one possible implementation, the AI ​​model includes a self-attention block, a first linear layer including a matrix multiplication operator for computing the output data of the self-attention block, and a second linear layer including a matrix multiplication operator for computing the value matrix in the input data of the self-attention block.

[0156] In one possible implementation, the first linear layer and the second linear layer are based on the GQA (Group Query Attention) architecture. The input data of the first linear layer is obtained by concatenating the output data of multiple heads in the GQA architecture. The second linear layer includes matrix multiplication operators in multiple heads for calculating the value matrix.

[0157] Update module 1002, used for:

[0158] Based on the first smoothing scale, the fourth smoothing scale is calculated. The value of each dimension in the fourth smoothing scale is obtained by calculating the values ​​of multiple dimensions in the first smoothing scale. The number of dimensions of the first smoothing scale is the same as the number of dimensions of the input data of the first linear layer, and the number of dimensions of the fourth smoothing scale is the same as the number of parameter dimensions of the matrix multiplication operator.

[0159] Update the parameter values ​​of the matrix multiplication operator based on the fourth smoothing scale.

[0160] In one possible implementation, the determining module 1001 is configured to:

[0161] Use AI models to infer from a calibrated dataset;

[0162] During the process of AI model reasoning based on the dataset, the input data for the first linear layer is determined;

[0163] The first smoothing scale is calculated based on the input data of the first linear layer and the parameter values ​​in the first linear layer.

[0164] Since the AI ​​model quantization device 1000 shown in Figure 10 corresponds to the quantization device 200 in the embodiment shown in Figure 5 above, the specific implementation method and technical effects of the AI ​​model quantization device 1000 shown in Figure 10 can be found in the relevant descriptions in the embodiment shown in Figure 5 above, and will not be repeated here.

[0165] For example, the AI ​​model quantization device 1000 described above can be implemented in software or hardware. In the first example, the AI ​​model quantization device 1000 can be implemented in software. In this case, the AI ​​model quantization device 1000 can specifically be code running on a computing instance, such as code running in a physical device, virtual machine, or container. In the second example, when implemented in hardware, the AI ​​model quantization device 1000 can be implemented using a processor, or using a physical device including a processor, such as a server. The processor can be a CPU, ASIC, PLD, CPLD, FPGA, GAL, SoC, SDI chip, AI chip, or DPU, or any combination of the above processors. Furthermore, the number of processors included in the AI ​​model quantization device 800 can be one or more, and the types of processors included can be one or more. The specific number and types of processors can be set according to the actual business requirements of the application, and this embodiment does not limit this.

[0166] Figure 11 is a schematic diagram of the hardware structure of a computing device 1100 provided in this application. The computing device 1100 can, for example, implement the quantization device 200 in the embodiment shown in Figure 5 above.

[0167] As shown in Figure 11, the computing device 1100 includes a processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, memory 1102, and communication interface 1103 communicate via a bus 1104, or via wireless transmission or other means. The memory 1102 stores instructions, and the processor 1101 executes the instructions stored in the memory 1102. Further, the computing device 1100 may also include a memory unit 1105, which is connected to the processor 1101, the storage medium 1102, and the communication interface 1103 via the bus 1104. The memory 1102 stores program code, and the processor 1101 can read the program code stored in the memory 1102 into the memory unit 1105 and execute the program code in the memory unit 1105 to perform the following operations:

[0168] A first smoothing scale is determined for the first linear layer in the AI ​​model. The first smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the first linear layer. The AI ​​model includes multiple consecutive network layers, including a first linear layer and a second linear layer. There is at least one network layer between the first linear layer and the second linear layer. The computation order of the second linear layer in the inference process of the AI ​​model is before the computation order of the first linear layer. The output data of the second linear layer is passed to the first linear layer through linear computation.

[0169] The parameter values ​​in the first linear layer and the parameter values ​​in the second linear layer are updated using the first smoothing scale;

[0170] After the parameter values ​​of at least one linear layer in the AI ​​model have been updated, the AI ​​model is quantized.

[0171] It should be understood that in this embodiment, the processor 1101 can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0172] The memory 1102 may include read-only memory and random access memory, and provides instructions and data to the processor 1101. The memory 1102 may also include non-volatile random access memory.

[0173] The memory 1102 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0174] The communication interface 1103 is used to communicate with other devices connected to the computing device 1100. The bus 1104 may include a data bus, as well as a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 1104 in the figure.

[0175] It should be understood that the computing device 1100 in this application embodiment can correspond to the method executed by the quantization device 200 in the method shown in FIG5 of this application embodiment. The above and other operations and / or functions implemented by the computing device 1100 are respectively to implement the flow of the corresponding method in FIG5. For the sake of brevity, they will not be described in detail here.

[0176] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned AI model quantization method.

[0177] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0178] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0179] The computer program product can be a software installation package. When any of the aforementioned AI model quantization methods is required, the computer program product can be downloaded and executed on a computing device.

[0180] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0181] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment. The terms “first,” “second,” etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate, and this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0182] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0183] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for quantifying artificial intelligence (AI) models, characterized in that, The method includes: A first smoothing scale is determined for the first linear layer in the AI ​​model. The first smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the first linear layer. The AI ​​model includes a plurality of consecutive network layers, including the first linear layer and a second linear layer. The first linear layer and the second linear layer are separated by at least one network layer. The computation order of the second linear layer in the inference process of the AI ​​model is before the computation order of the first linear layer. The output data of the second linear layer is passed to the first linear layer through linear computation. The parameter values ​​in the first linear layer and the parameter values ​​in the second linear layer are updated using the first smoothing scale; After at least one linear layer in the AI ​​model has completed parameter value updates, the AI ​​model is quantized.

2. The method according to claim 1, characterized in that, The plurality of network layers also include a normalization layer, wherein the normalization layer is computed before the second linear layer in the AI ​​model inference process; The method further includes: After updating the parameter values ​​in the second linear layer using the first smoothing scale, a second smoothing scale is determined for the second linear layer. The second smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the second linear layer. Before quantizing the AI ​​model, the parameter values ​​in the second linear layer and the parameter values ​​in the normalization layer are updated using the second smoothing scale.

3. The method according to claim 2, characterized in that, The plurality of network layers further includes a third linear layer, the input data of which is the same as the input data of the second linear layer; The method further includes: Before quantizing the AI ​​model, a third smoothing scale is determined for the third linear layer. The third smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the third linear layer. Updating the parameter values ​​in the second linear layer and the parameter values ​​in the normalization layer using the second smoothing scale includes: Calculate the average smoothing scale between the second smoothing scale and the third smoothing scale; The parameter values ​​in the normalized layer, the parameter values ​​in the second linear layer, and the parameter values ​​in the third linear layer are updated using the average smoothing scale.

4. The method according to claim 2 or 3, characterized in that, The plurality of network layers further includes a fourth linear layer, wherein the computation order of the fourth linear layer precedes the computation order of the normalization layer; the method further includes: Without including a normalization layer and a linear layer with parameter values ​​before the calculation order of the fourth linear layer, a smoothing operator is added to the fourth linear layer to reduce the quantization error caused by the parameter values ​​in the fourth linear layer.

5. The method according to any one of claims 2 to 4, characterized in that, The AI ​​model includes a multilayer perceptron (MLP) block, which includes a normalization layer, a first linear layer, and a second linear layer. The output of the normalization layer is the input of the second linear layer.

6. The method according to claim 1, characterized in that, The AI ​​model includes a self-attention block, the first linear layer includes a matrix multiplication operator for calculating the output data of the self-attention block, and the second linear layer includes a matrix multiplication operator for calculating the value matrix in the input data of the self-attention block.

7. The method according to claim 1, characterized in that, The first linear layer and the second linear layer are based on the Group Query Attention (GQA) architecture. The input data of the first linear layer is obtained by concatenating the output data of multiple heads in the GQA architecture. The second linear layer includes matrix multiplication operators in the multiple heads for calculating the value matrix. Updating the parameter values ​​in the second linear layer using the first smoothing scale includes: Based on the first smoothing scale, a fourth smoothing scale is calculated. The value of each dimension in the fourth smoothing scale is obtained by calculating the values ​​of multiple dimensions in the first smoothing scale. The number of dimensions of the first smoothing scale is the same as the number of dimensions of the input data of the first linear layer, and the number of dimensions of the fourth smoothing scale is the same as the number of parameter dimensions of the matrix multiplication operator. The parameter values ​​of the matrix multiplication operator are updated according to the fourth smoothing scale.

8. The method according to any one of claims 1 to 7, characterized in that, Determining the first smoothing scale for the first linear layer in the AI ​​model includes: The AI ​​model is used to perform inference on the calibrated dataset; During the process of the AI ​​model performing inference based on the dataset, the input data of the first linear layer is determined; The first smoothing scale is calculated based on the input data of the first linear layer and the parameter values ​​in the first linear layer.

9. An AI model quantization device, characterized in that, The device includes: A determination module is used to determine a first smoothing scale for a first linear layer in an AI model. The first smoothing scale is used to reduce the quantization error caused by the parameter values ​​in the first linear layer. The AI ​​model includes a plurality of consecutive network layers, including the first linear layer and a second linear layer. The first linear layer and the second linear layer are separated by at least one network layer. The calculation order of the second linear layer in the inference process of the AI ​​model is before the calculation order of the first linear layer. The output data of the second linear layer is passed to the first linear layer through linear calculation. The update module is used to update the parameter values ​​in the first linear layer and the parameter values ​​in the second linear layer using the first smoothing scale; The quantization module is used to quantize the AI ​​model after at least one linear layer in the AI ​​model has completed parameter value updates.

10. The apparatus according to claim 9, characterized in that, The plurality of network layers also include a normalization layer, wherein the normalization layer is computed before the second linear layer in the AI ​​model inference process; The determining module is further configured to determine a second smoothing scale for the second linear layer after updating the parameter values ​​in the second linear layer using the first smoothing scale, the second smoothing scale being used to reduce the quantization error caused by the parameter values ​​in the second linear layer; The update module is further configured to update the parameter values ​​in the second linear layer and the parameter values ​​in the normalization layer using the second smoothing scale before quantizing the AI ​​model.

11. The apparatus according to claim 10, characterized in that, The plurality of network layers further includes a third linear layer, the input data of which is the same as the input data of the second linear layer; The determining module is further configured to determine a third smoothing scale for the third linear layer before quantizing the AI ​​model, the third smoothing scale being used to reduce the quantization error caused by the parameter values ​​in the third linear layer; The update module is used for: Calculate the average smoothing scale between the second smoothing scale and the third smoothing scale; The parameter values ​​in the normalized layer, the parameter values ​​in the second linear layer, and the parameter values ​​in the third linear layer are updated using the average smoothing scale.

12. The apparatus according to claim 10 or 11, characterized in that, The plurality of network layers also includes a fourth linear layer, the fourth linear layer being computed before the normalization layer being computed. The apparatus further includes an adding module, configured to add a smoothing operator to the fourth linear layer when the normalization layer and the linear layer with parameter values ​​are not included before the calculation order of the fourth linear layer, the smoothing operator being used to reduce the quantization error caused by the parameter values ​​in the fourth linear layer.

13. The apparatus according to any one of claims 10 to 12, characterized in that, The AI ​​model includes a multilayer perceptron (MLP) block, which includes a normalization layer, a first linear layer, and a second linear layer. The output of the normalization layer is the input of the second linear layer.

14. The apparatus according to claim 9, characterized in that, The AI ​​model includes a self-attention block, the first linear layer includes a matrix multiplication operator for calculating the output data of the self-attention block, and the second linear layer includes a matrix multiplication operator for calculating the value matrix in the input data of the self-attention block.

15. The apparatus according to claim 9, characterized in that, The first linear layer and the second linear layer are based on the Group Query Attention (GQA) architecture. The input data of the first linear layer is obtained by concatenating the output data of multiple heads in the GQA architecture. The second linear layer includes matrix multiplication operators in the multiple heads for calculating the value matrix. The update module is used for: Based on the first smoothing scale, a fourth smoothing scale is calculated. The value of each dimension in the fourth smoothing scale is obtained by calculating the values ​​of multiple dimensions in the first smoothing scale. The number of dimensions of the first smoothing scale is the same as the number of dimensions of the input data of the first linear layer, and the number of dimensions of the fourth smoothing scale is the same as the number of parameter dimensions of the matrix multiplication operator. The parameter values ​​of the matrix multiplication operator are updated according to the fourth smoothing scale.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The determining module is used for: The AI ​​model is used to perform inference on the calibrated dataset; During the process of the AI ​​model performing inference based on the dataset, the input data of the first linear layer is determined; The first smoothing scale is calculated based on the input data of the first linear layer and the parameter values ​​in the first linear layer.

17. A computing device, characterized in that, The computing device includes a processor and memory; The memory is used to store instructions, and the processor executes the instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1 to 8.

19. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 8.

Images