Method for determining output result of model, and related apparatus

Abstract

The present application belongs to the technical field of data processing. Disclosed are a method for determining an output result of a model, and a related apparatus. A network model comprises an attention mechanism module. The method comprises: on the basis of current input information of a network model, determining a query matrix, a key matrix and a value matrix by means of an attention mechanism module; processing a transposed matrix of the key matrix, so as to obtain an intermediate matrix, wherein the intermediate matrix has the same number of rows and columns as those in the transposed matrix, and the average value of elements in each column in the intermediate matrix is less than the average value of elements in a corresponding column in the transposed matrix; on the basis of the query matrix, the intermediate matrix and the value matrix, determining an output matrix by means of the attention mechanism module; and on the basis of the output matrix, determining a current output result of the network model. It can be seen therefrom that in the present application, a transposed matrix of a key matrix is processed, so as to decrease the values of elements in the transposed matrix, thereby preventing a computational source from the problems of numerical overflow and computational instability.

Description

Methods and related devices for determining model output results

[0001] This application claims priority to Chinese Patent Application No. 202411719646.5, filed on November 26, 2024, entitled "Method and Related Apparatus for Determining Model Output Results", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data processing technology, and in particular to a method and related apparatus for determining model output results. Background Technology

[0003] In the inference process of large-scale deep learning models (also known as large models), the attention mechanism plays a crucial role. It allows large models to dynamically focus on different parts of the input information when processing it, thereby capturing richer and more complex information. However, the computational complexity of the attention mechanism is high. For low-precision data types (such as 16-bit or 8-bit floating-point numbers), when the attention mechanism determines the output matrix, the relatively small representation range of low-precision data types can lead to numerical overflow when intermediate or final results exceed the representation range of the data type. This can affect the accuracy and stability of the network model. Summary of the Invention

[0004] This application provides a method and related apparatus for determining model output results, which can prevent numerical overflow of the attention mechanism during the processing of input information in large models. The technical solution is as follows:

[0005] Firstly, a method for determining the output of a model is provided, wherein the network model includes an attention mechanism module; the method includes:

[0006] Based on the current input information of the network model, the attention mechanism module determines the query matrix, key matrix, and value matrix; the transpose of the key matrix is ​​processed to obtain an intermediate matrix, which has the same number of rows and columns as the transpose matrix, and the average value of each column element in the intermediate matrix is ​​less than the average value of the corresponding column elements in the transpose matrix; based on the query matrix, the intermediate matrix, and the value matrix, the attention mechanism module determines the output matrix; based on the output matrix, the current output result of the network model is determined.

[0007] Therefore, for network models including attention mechanisms, the technical solution of this application processes the key matrix before performing matrix operations to obtain an intermediate matrix. Then, using the query matrix, intermediate matrix, and value matrix, the attention mechanism module determines the output matrix. Since the average value of each column element in the intermediate matrix is ​​less than the average value of the corresponding column elements in the transpose of the key matrix (i.e., the values ​​of elements in the intermediate matrix are attenuated relative to the transpose of the key matrix), numerical overflow and computational instability can be prevented at the computational source.

[0008] In one possible implementation, processing the transpose of the key matrix to obtain the intermediate matrix includes:

[0009] Obtain the offset parameter; construct an offset matrix based on the offset parameter and the number of columns of the transpose matrix, wherein all elements in the offset matrix are less than 1, the number of columns in the offset matrix is ​​the same as that in the transpose matrix, and the number of rows and columns in the offset matrix are equal; perform matrix multiplication on the transpose matrix and the offset matrix to obtain the intermediate matrix.

[0010] In one possible implementation, the transpose matrix comprises multiple transpose basic blocks, each containing elements with multiple rows and columns, and the elements in each transpose basic block have the same number of rows and columns; the process of processing the transpose matrix of the key matrix to obtain an intermediate matrix includes:

[0011] Obtain offset parameters; construct an offset matrix based on the offset parameters and the number of columns of the elements included in the transposed basic block, wherein each element in the offset matrix is ​​less than 1, the number of columns of the offset matrix is ​​the same as that of the elements included in the transposed basic block, and the number of rows and columns of the offset matrix are equal; perform matrix multiplication operations between the multiple transposed basic blocks and the offset matrix respectively to obtain multiple intermediate basic blocks; generate the intermediate matrix based on the multiple intermediate basic blocks.

[0012] Therefore, when processing the transpose of the key matrix, this application can process the entire matrix or process the transpose basic blocks of the transpose matrix separately. The processing granularity of the two methods is different, but both can achieve the effect of attenuating the values ​​of the elements in the transpose matrix, thereby reducing the possibility of numerical overflow at the source.

[0013] In one possible implementation, the elements of the main diagonal of the offset matrix are: The elements not on the main diagonal are Wherein, α is the offset parameter, and s2 is the number of rows or columns of the offset matrix.

[0014] Due to the main diagonal elements in the offset matrix Since these are positive numbers, during matrix multiplication of the offset matrix and its transpose, these diagonal elements are multiplied by the corresponding elements in the transpose matrix, thus preserving some of the values ​​of these elements in the transpose matrix (although multiplied by a coefficient less than 1). However, this effect is local, limited to the positions in the transpose matrix corresponding to the diagonal elements of the offset matrix. This is because the non-diagonal elements of the offset matrix... The negative values ​​of these off-diagonal elements have a "decaying" effect on the elements of the transpose matrix during matrix multiplication. Specifically, when an element of the transpose matrix is ​​multiplied by the off-diagonal elements of the offset matrix, the result is negative, and the absolute value of this negative number depends on the ratio of α and s². Furthermore, this negative product is added to other products (including products with the off-diagonal elements of the offset matrix), potentially reducing the value of the corresponding element in the transpose matrix.

[0015] Therefore, since all elements in the offset matrix are less than 1, and the elements on the main diagonal are... The elements not on the main diagonal are Therefore, when performing matrix multiplication between the offset matrix and the transpose matrix, the diagonal elements in the offset matrix will retain the values ​​of the corresponding elements in the transpose matrix to a certain extent, while the off-diagonal elements will have a significant attenuation effect on the elements in the transpose matrix. This effect will greatly reduce the values ​​of the elements in the transpose matrix during matrix multiplication, thereby preventing numerical overflow and computational instability at the source of attention computation.

[0016] In one possible implementation, the query matrix includes multiple query basic blocks, the intermediate matrix includes multiple intermediate basic blocks, and the value matrix includes multiple value basic blocks; the step of determining the output matrix based on the query matrix, the intermediate matrix, and the value matrix through the attention mechanism module includes:

[0017] Matrix multiplication is performed on the multiple query basic blocks and the multiple intermediate basic blocks to obtain multiple fractional basic blocks; multiple weight basic blocks are determined based on the multiple fractional basic blocks; multiple output basic blocks are determined based on the multiple weight basic blocks and the multiple value basic blocks; offset correction is performed on the multiple output basic blocks to obtain corrected multiple output basic blocks; the output matrix is ​​generated based on the corrected multiple output basic blocks.

[0018] In the standard attention mechanism, the computation process involves matrix multiplication of the entire matrix followed by normalization using the softmax function. Because the softmax function is translation-invariant, the correctness of the final output matrix is ​​not affected even if the intermediate matrix is ​​obtained by processing the transpose of the key matrix. However, in the FA (Automatic Attention) computation process, matrix multiplication is performed on a block-by-block basis, followed by softmax normalization. Since the average value of each block may differ, the processing results may be biased, leading to a bias in the output matrix. Therefore, in the block-based approach, the output blocks need to be corrected to obtain the correct output matrix.

[0019] In one possible implementation, the plurality of weight basic blocks are distributed in M ​​rows and N columns, where M and N are both integers greater than or equal to 1. The plurality of weight basic blocks have the same number of rows of elements, the plurality of output basic blocks have the same number of rows of elements, and the weight basic blocks and output basic blocks also have the same number of rows of elements. The offset correction of the plurality of output basic blocks to obtain corrected plurality of output basic blocks includes:

[0020] Determine multiple global row sums for the weight basic block in the k-th row of the multiple weight basic blocks, where the global row sum is the sum of a row of elements in the weight basic block in the k-th row, k is greater than or equal to 1 and less than or equal to M; correct the i-th row element of the output basic block in the k-th row of the multiple output basic blocks to obtain the corrected i-th row element of the output basic block in the k-th row, where i is greater than or equal to 1 and less than or equal to the number of rows of elements included in the output basic block; divide the corrected i-th row element of the output basic block in the k-th row by the corresponding global row sum in the multiple global row sums to obtain the corrected output basic block in the k-th row.

[0021] Where M represents the number of query base blocks and N represents the number of intermediate base blocks. That is, for each fractional base block, the number of rows in the fractional base block is equal to the number of rows in the corresponding query base block; the number of columns in the fractional base block is equal to the number of columns in the corresponding intermediate base block.

[0022] In one possible implementation, the plurality of value basic blocks are distributed in N rows, the plurality of value basic blocks include elements with the same number of columns, the plurality of output basic blocks include elements with the same number of columns, and the value basic blocks and the output basic blocks include elements with the same number of columns; the step of modifying the i-th row element of the k-th row of the output basic block to obtain the modified i-th row element of the k-th row of the output basic block includes:

[0023] Let j = 1. Determine the correction parameter for the row sum of the elements in the i-th row of the first j-1 weight basic blocks in the k-th row, and the correction parameter for the row sum of the elements in the i-th row of the j-th weight basic block, where the row sum is the sum of the elements in the i-th row. Obtain the first and second values ​​of the Oig element in the output basic block of the k-th row, where the Oig element refers to the element located in the i-th row and g-th column of the output basic block of the k-th row. The first value is determined based on the elements in the i-th row of the first j-1 weight basic blocks and the elements in the g-th column of the value basic block of the k-th row. The second value is determined based on the elements in the i-th row of the j-th weight basic block and the elements in the g-th column of the value basic block of the k-th row, where g is greater than or equal to 1 and less than 1. The value is equal to or greater than the number of columns of elements included in the value basic block; based on the first value, the second value, the correction parameter of the row sum of the i-th row elements of the first j-1 weight basic blocks, and the correction parameter of the row sum of the i-th row elements of the j-th weight basic block, the third value of the Oig element in the output basic block of the k-th row is determined; if j = N, the third value of the Oig element in the output basic block of the k-th row is determined as the corrected value of the i-th row element of the output basic block of the k-th row; if j < N, j = j + 1, and the steps of determining the correction parameter of the row sum of the i-th row elements of the first j-1 weight basic blocks and the correction parameter of the row sum of the i-th row elements of the j-th weight basic block are returned.

[0024] In one possible implementation, determining the global row sums of the weighted basic blocks in the k-th row of the plurality of weighted basic blocks includes:

[0025] Let j = 1, determine the correction parameter for the row sum of the elements in the i-th row of the first j-1 weight basic blocks in the k-th row, and the correction parameter for the row sum of the elements in the i-th row of the j-th weight basic block, where the row sum is the sum of the elements in the i-th row; obtain the row sum of the elements in the i-th row of the first j-1 weight basic blocks, and the row sum of the elements in the i-th row of the j-th weight basic block; based on the correction parameter for the row sum of the elements in the i-th row of the first j-1 weight basic blocks, the row sum of the elements in the i-th row of the first j-1 weight basic blocks, and the row sum of the elements in the i-th row of the j-th weight basic block... The steps are as follows: First, determine the corrected row sum of the i-th row elements of the first j weight basic blocks in the k-th weight basic block, using the correction parameters and the row sum of the i-th row elements of the j-th weight basic block. If j = N, then determine the corrected row sum of the i-th row elements of the first j weight basic blocks as the global row sum of the i-th row elements of the first j weight basic blocks. If j < N, then let j = j + 1, and return to the steps of determining the correction parameters for the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th weight basic block, and the correction parameters for the row sum of the i-th row elements of the j-th weight basic block.

[0026] In one possible implementation, the plurality of weight basic blocks correspond one-to-one with the plurality of fraction basic blocks, the plurality of fraction basic blocks include the same number of rows of elements, and the fraction basic blocks and weight basic blocks also include the same number of rows of elements; the step of determining the correction parameters for the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th row weight basic block, and the correction parameters for the row sum of the i-th row elements of the j-th weight basic block, includes:

[0027] Obtain correction parameters for the maximum value of the i-th row element of the first j-1 fractional basic blocks in the k-th row of the plurality of fractional basic blocks, and correction parameters for the maximum value of the i-th row element of the j-th fractional basic block; obtain the maximum value of the i-th row element of the first j-1 fractional basic blocks, the maximum value of the i-th row element of the j-th fractional basic block, and the maximum value of the i-th row element of the first j-1 fractional basic blocks; based on the correction parameters for the maximum value of the i-th row element of the first j-1 fractional basic blocks, the maximum value of the i-th row element of the first j-1 fractional basic blocks, and the maximum value of the i-th row element of the first j-1 fractional basic blocks, determine correction parameters for the row sum of the i-th row elements of the first j-1 weighted basic blocks; based on the correction parameters for the maximum value of the i-th row element of the j-th fractional basic block, the maximum value of the i-th row element of the first j-1 fractional basic blocks, and the maximum value of the i-th row element of the j-th fractional basic block, determine correction parameters for the row sum of the i-th row elements of the j-th weighted basic block.

[0028] In one possible implementation, the correction parameter for obtaining the maximum value of the i-th row element of the first j-1 fractional basic blocks in the k-th row of the plurality of fractional basic blocks, and the correction parameter for the maximum value of the i-th row element of the j-th fractional basic block, includes:

[0029] Obtain the average value of the i-th row element of the first j-1 fractional basic blocks, the average value of the i-th row element of the j-th fractional basic block, and the average value of the i-th row element of the first j-1 fractional basic blocks; obtain the offset parameter; based on the average value of the i-th row element of the first j-1 fractional basic blocks, the average value of the i-th row element of the j-th fractional basic block, the average value of the i-th row element of the first j-1 fractional basic blocks, and the offset parameter, determine the correction parameter for the maximum value of the i-th row element of the first j-1 fractional basic blocks and the correction parameter for the maximum value of the i-th row element of the j-th fractional basic block.

[0030] In one possible implementation, obtaining the average of the elements in the i-th row of the first j fractional basic blocks includes:

[0031] The average value of the i-th row of the first j-1 fractional basic blocks is determined based on the average value of the i-th row of the first j-1 fractional basic blocks and the average value of the i-th row of the j-th fractional basic block.

[0032] In one possible implementation, obtaining the maximum value of the i-th row element of the first j basic fraction blocks includes:

[0033] Based on the correction parameter of the maximum value of the i-th row element of the first j-1 fractional basic blocks, the maximum value of the i-th row element of the first j-1 fractional basic blocks is corrected to obtain the corrected maximum value of the i-th row element of the first j-1 fractional basic blocks; based on the correction parameter of the maximum value of the i-th row element of the j-th fractional basic block, the maximum value of the i-th row element of the j-th fractional basic block is corrected to obtain the corrected maximum value of the i-th row element of the j-th fractional basic block; from the corrected maximum value of the i-th row element of the first j-1 fractional basic blocks and the corrected maximum value of the i-th row element of the j-th fractional basic block, the maximum value of the i-th row element of the first j-1 fractional basic blocks is determined.

[0034] Therefore, by modifying the output basic block of the k-th row, this application can eliminate the deviation introduced when the transpose of the key matrix was processed using the offset matrix in the early stage, thereby ensuring the correctness of the attention calculation results.

[0035] In one possible implementation, obtaining the offset parameter includes:

[0036] Obtain the data type of any one of the query matrix, the key matrix, and the value matrix, wherein the data type indicates the floating-point precision of the data; based on the data type, determine the offset parameter from the correspondence between the data type and the offset parameter.

[0037] In other words, this application can configure the offset parameter with optimal precision based on the input data of the FA operator during network model inference, namely the data type of the query matrix and the key-value cache sequence, so as to ensure the minimum error under low precision (e.g., 16-bit floating-point number, 8-bit floating-point number) calculation.

[0038] Secondly, an apparatus for determining model output results is provided, the apparatus having the function of implementing the method behavior of determining model output results in the first aspect described above. The apparatus for determining model output results includes at least one module, which is used to implement the method for determining model output results provided in the first aspect.

[0039] Thirdly, a computer device is provided, comprising a processor and a memory, the memory being used to store a computer program for performing the method for determining model output results provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the method for determining model output results as described in the first aspect.

[0040] Optionally, the computer device may further include a communication bus for establishing a connection between the processor and the memory.

[0041] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the method for determining the output result of the model as described in the first aspect.

[0042] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the method for determining the output result of a model as described in the first aspect. Alternatively, a computer program is provided that, when executed on a computer, causes the computer to perform the method for determining the output result of a model as described in the first aspect.

[0043] The technical effects achieved by the second, third, fourth, and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0044] Figure 1 is a schematic diagram of an Attention calculation process provided in an embodiment of this application;

[0045] Figure 2 is a schematic diagram of the calculation process of an FA operator provided in an embodiment of this application;

[0046] Figure 3 is a schematic diagram of a GPU hardware architecture and data path provided in an embodiment of this application;

[0047] Figure 4 is a schematic diagram of the hardware architecture and data path of an NPU provided in an embodiment of this application;

[0048] Figure 5 is a schematic diagram of a large model inference scenario provided in an embodiment of this application;

[0049] Figure 6 is a schematic diagram of another large model inference scenario provided by an embodiment of this application;

[0050] Figure 7 is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0051] Figure 8 is a flowchart illustrating a method for determining model output results provided in an embodiment of this application;

[0052] Figure 9 is a schematic diagram of a process for generating an intermediate matrix according to an embodiment of this application;

[0053] Figure 10 is a flowchart illustrating the process of determining the output matrix using an attention mechanism module according to an embodiment of this application.

[0054] Figure 11 is a schematic flowchart of a method for determining a fractional basic block according to an embodiment of this application;

[0055] Figure 12 is a flowchart illustrating the process of determining a basic weight block according to an embodiment of this application;

[0056] Figure 13 is a flowchart illustrating the process of determining the row and correction parameters of each row element of the k-th row weighted basic block according to an embodiment of this application.

[0057] Figure 14 is a flowchart illustrating another method for determining model output results provided in an embodiment of this application;

[0058] Figure 15 is a schematic diagram of the structure of a device for determining model output results provided in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0060] To facilitate understanding, before explaining the method for determining the output result of the model provided in this application, the terminology, application scenarios, and implementation environment involved in the embodiments of this application will be introduced first.

[0061] First, the relevant terminology in the embodiments of this application will be introduced.

[0062] 1. Large-scale model reasoning

[0063] Large models (also known as foundation models) are machine learning models with a large number of parameters and complex computational structures. These models are typically built from deep neural networks and have billions or even hundreds of billions of parameters. The purpose of large models is to improve their expressive power and predictive performance, enabling them to handle more complex tasks and data. Currently, large models have wide applications in various fields, including but not limited to natural language processing, computer vision, speech recognition, and recommender systems.

[0064] Based on the different types of input data, large models can be divided into three main categories: language large models, vision large models, and multimodal large models. Language large models refer to a class of large models in the field of natural language processing (NLP), typically used to process text data and understand natural language. The main characteristic of these large models is that they are trained on large-scale corpora to learn various grammatical, semantic, and contextual rules of natural language. Vision large models refer to large models used in the field of computer vision (CV), typically used for image processing and analysis. These models, trained on large-scale image data, can perform various visual tasks, such as image classification, object detection, image segmentation, pose estimation, and face recognition. Multimodal large models are large models capable of processing multiple different types of data, such as text, images, and audio. These models combine NLP and CV capabilities to achieve a comprehensive understanding and analysis of multimodal information, thus enabling a more comprehensive understanding and processing of complex data.

[0065] The Transformer architecture, as one of the most popular large-scale model architectures, achieves efficient processing of sequence data through attention mechanisms, yielding significant results in the field of large models. These attention mechanisms include various forms such as self-attention and cross-attention. Self-attention allows the model to focus on different positions within a sequence simultaneously, capturing dependencies within the sequence; while cross-attention is used to handle dependencies between two sequences, such as the alignment between source and target languages ​​in machine translation. The attention mechanism in the Transformer architecture calculates relevance scores between different positions in the input sequence, then performs a weighted sum of these scores to obtain the output sequence. This computational approach enables the Transformer architecture to handle input sequences of infinite length while maintaining high efficiency and accuracy during computation.

[0066] 2. Attention Mechanism

[0067] Referring to Figure 1, the Attention calculation process typically involves three key matrices: the Query (Q) matrix, the Key (K) matrix, and the Value (V) matrix. The dimensions of these three matrices are Q∈R. S1×d K∈R S2×d and V∈R S2×dS1 and S2 represent the lengths of the input sequences, which may be equal or unequal, and d is the feature dimension. The Attention calculation process may include the following steps:

[0068] (1) Calculate the fraction matrix.

[0069] Calculate the transpose (K) of the query matrix and the key matrix. T The dot product of the input sequence (i.e., matrix multiplication) yields a fractional matrix (denoted as S), which has dimensions S1×S2. This fractional matrix reflects the correlation between different parts of the input sequence.

[0070] The mathematical formula for calculating the fractional matrix is: S = QK T ∈R S1×S2 .

[0071] (2) Scaling the fractional matrix.

[0072] In practical calculations, to maintain numerical stability, the fractional matrix is ​​usually scaled, i.e., divided by a constant σ.

[0073] The mathematical formula for scaling the fractional matrix is: S = S / σ∈R S1×S2 , d refers to the feature dimension described above.

[0074] (3) Apply the softmax function.

[0075] Applying the softmax function to the fraction matrix yields a weight matrix (i.e., a probability distribution matrix, denoted as P), which still has dimensions S1×S2. This probability distribution matrix reflects the relative importance of different parts of the input sequence.

[0076] The mathematical formula for calculating the weight matrix is: P = softmax(S) ∈ R S1×S2 .

[0077] (4) Calculate the weighted sum.

[0078] Performing matrix multiplication on the probability distribution matrix P and the value matrix V yields an output matrix (denoted as O) with dimensions S1×d. This output matrix contains feature information from different parts of the input sequence weighted according to their relative importance.

[0079] The mathematical formula for calculating the output matrix is: O = PV ∈ R S1×d .

[0080] In the aforementioned Attention standard computation process, the precision of different data types is often involved. For example, the calculation of the fractional matrix S typically uses 16-bit floating-point (fp16) data type to save computational resources and improve calculation speed. However, during the calculation, it may be temporarily upgraded to 32-bit floating-point (fp32) precision to maintain numerical stability. Scaling operations, the softmax function, and weighted sum calculations typically use fp32 data type to ensure the accuracy and stability of the calculation.

[0081] Furthermore, it should be noted that the weight matrix P is not shown in Figure 1 above, but it should be understood that the weight matrix P and the fraction matrix S have the same dimension and their elements are also in one-to-one correspondence. The only difference is that the weight matrix P is a normalized matrix, and the values ​​of the elements in this matrix are in the range of 0-1.

[0082] 3. Flash attention (FA)

[0083] Based on the standard Attention computation process, Dr. Tri Dao, a large model expert at Stanford University, released a series of flash attention (FA) models. The evolution process of the FA operator is shown in Table 1 below.

[0084] Table 1

[0085] It's important to note that the FA operator is essentially an algorithm optimization technique that leverages the memory hierarchy of underlying hardware, such as the memory structures of central processing units (CPUs), graphics processing units (GPUs), and neural network processing units (NPUs), to improve computational speed and reduce memory access overhead. In other words, the FA operator is not a traditional hardware operator, but rather a software-level optimization method.

[0086] Compared to the standard Attention computation process described above, the FA operator employs a chunking algorithm and online softmax update. These optimizations enable the FA operator to handle theoretically unconstrained sequence lengths, while significantly improving computational performance and substantially reducing the use of high bandwidth memory (HBM).

[0087] In the standard Attention computation process, a score matrix S with dimensions S1×S2 is typically calculated, where S1 and S2 represent the lengths of the input sequence. When the sequence length is very long, this score matrix becomes extremely large, leading to a sharp increase in computational resources and memory usage. To address this, the FA operator introduces a block-based algorithm. The basic idea of ​​this algorithm is to divide the input sequence into multiple smaller chunks and then perform Attention computation on each chunk separately. This significantly reduces the computational load and memory usage within each chunk, while also allowing for parallel processing of multiple chunks using multi-level memory structures such as GPUs.

[0088] Furthermore, in the standard Attention computation process, the softmax function is typically used to transform the score matrix S into a weight matrix P. When the score matrix S is very large, the computational cost and memory usage of the softmax function become extremely high. Therefore, the FA operator addresses this issue by introducing online softmax updates. The basic idea of ​​online softmax updates is to calculate the softmax value of each element step-by-step, rather than calculating the softmax of the entire score matrix S all at once, and dynamically updating the relevant probability distribution. This online softmax update approach not only reduces memory usage but also improves computational efficiency.

[0089] Referring to Figure 2, in the FA calculation process, the FA operator divides the Q, K, and V matrices of the input sequence into multiple basic blocks. Then, for each basic block, the following operations are performed: Calculate the fractional matrix S_chunk within the basic block; perform scaling and softmax operations on the fractional matrix S_chunk to obtain the weight matrix P_chunk; perform a weighted sum calculation using the weight matrix P_chunk and the corresponding value matrix V_chunk to obtain the output matrix O_chunk; finally, concatenate the output matrices O_chunk of all basic blocks to obtain the final output matrix O. Specifically, when calculating the weight matrix P_chunk of a basic block, the FA operator uses an equivalent method of subtracting the maximum value of the fractional matrix to avoid directly handling large number overflow issues during softmax calculation. Furthermore, by utilizing the pipelined structure and multi-level memory structure of the underlying hardware, the FA operator can progressively calculate the softmax value of each basic block and dynamically update the relevant probability distribution during the calculation process.

[0090] It should be noted that the weight matrix P is not shown in Figure 2 above, but it should be understood that the weight matrix P and the fraction matrix S have the same dimension and their elements are also in one-to-one correspondence. The only difference is that the weight matrix P is a normalized matrix, and the values ​​of the elements in this matrix are in the range of 0-1.

[0091] 4. Key-value caching (KV Cache) technology

[0092] KV Cache technology is primarily based on the self-attention mechanism in the Transformer model. In this mechanism, the model first computes three vector sets: query (Q), key (K), and value (V). These vectors are typically obtained by applying different linear transformations to the embedding vectors of the input sequence. Then, the model uses matrix multiplication to calculate the dot product of the query vector and all key vectors, generating a weight distribution. Finally, it uses a weighted average of these weights to obtain the corresponding value vector, thus capturing the contextual information at each position.

[0093] In generative models, only one new character / word (token) is generated at a time during inference. This new character / word is then concatenated with the previously generated portion of the input sequence to form a longer sequence, which is then fed back into the model to generate the next token. This autoregressive characteristic means that the model needs to repeatedly process the same or similar input sequence fragments, especially in multi-turn dialogues or long text generation scenarios. Since the model has to recalculate the K and V vectors of the entire input sequence during each inference, and these vectors are invariant for the same input token, this results in a large amount of redundant computation.

[0094] Based on this, the KV Cache technology caches previously calculated K and V vectors, which the model can directly reuse in subsequent inference steps, avoiding redundant calculations and significantly reducing the amount of computation required, thereby speeding up inference.

[0095] Key-value (KV) caching technology is widely used in large language models (LLMs) that require efficient inference. In scenarios such as text generation, question answering systems, and dialogue systems, KV caching can significantly improve the model's response speed and reduce computational costs.

[0096] 5. Operator library

[0097] The computational flow of FA (Functional Aspect Ratio) involves complex sets of operator operations, such as matrix multiplication (matmul) and softmax operations. Therefore, FA operators are also called fusion operators. Since FA operators are executed on hardware, hardware manufacturers also provide FA operator libraries optimized for their hardware to ensure the best performance on specific hardware.

[0098] In the field of artificial intelligence (AI), open-source operator libraries are a common practice. This helps cultivate an ecosystem, attracting more developers to use these operator libraries, and continuously optimizing and improving operator performance through community efforts. Therefore, various hardware manufacturers also consider FA (Automatic Facilitator) operator libraries a key part of their hardware product's technological competitiveness, attracting users by providing efficient and reliable operator libraries.

[0099] It should be understood that the FA operator library is a collection containing various FA operators. The existence of the FA operator library allows developers to easily call and use these predefined FA operators without having to implement them from scratch, greatly improving development efficiency, reducing development costs, and ensuring the consistency and portability of algorithms.

[0100] The FA operator library is an important component of hardware products. It provides the functions and algorithms required to perform calculations on the hardware. The performance and compatibility of the operator library directly affect users' choices and satisfaction with hardware products.

[0101] Next, we will use GPUs and NPUs as examples to introduce their hardware architecture for running FA operators. GPUs are processors specifically designed for parallel computing, while NPUs are processors specifically designed for deep learning computations. Both have multi-layered memory architectures and corresponding data pathways to achieve efficient parallel computing and data processing.

[0102] Referring to Figure 3, the multi-layered memory architecture of a GPU includes: dynamic random access memory (DRAM), level 2 cache (also known as L2 cache), shared memory (SMEM), and register file (RF). The main computing units of a GPU include: tensor computing units (also known as Tensor Cores) and compute unified device architecture (CUDA) computing units (also known as CUDA Cores).

[0103] DRAM is the GPU's main memory, used to store large amounts of data and instructions. It has high capacity but relatively slow access speed. Therefore, data usually needs to be loaded from DRAM into a faster memory tier before the GPU performs computations. L2 cache is the second layer of memory in the GPU, located between DRAM and SMEM. It has high access speed but relatively small capacity. In practical applications, L2 cache is used to store recently accessed data and instructions to reduce the number of DRAM accesses, thereby improving computational efficiency. SMEM is a shared memory region in the GPU, typically used to store shared data within thread blocks. It has high access speed and low latency, making it ideal for data sharing and communication between threads. RF is the register file in the GPU, used to store thread-local data and instructions. Each thread has its own registers, so the register file has high access speed and low latency. The register file is one of the most important memory tiers in the GPU because it is directly connected to the computation units, providing data and instructions quickly. Tensor Cores and CUDA Cores are the main computing units in the GPU, responsible for performing computational tasks. Data is transferred from DRAM through L2 cache, SMEM, and RF to these two computing units for processing.

[0104] Tensor Cores are high-performance computing units in GPUs specifically designed for deep learning computations. They support mixed-precision computing and can efficiently perform matrix multiplication and accumulation operations. CUDA Cores are based on the single-instruction multiple threads (SIMT) model, allowing multiple threads to execute simultaneously. Each thread can operate on different data, making CUDA Cores ideal for handling large-scale parallel computing tasks.

[0105] Referring to Figure 4, the multi-layered memory architecture of the NPU includes: global memory (GM), L2 cache, L1 cache, L0 cache, and unified buffer (UB). The main computing units of the NPU include: Cube computing units and Vector (Vec) computing units.

[0106] GM (General Storage Unit) is the large-capacity storage unit of the NPU, typically used to store neural network models and data. L2 cache, located between GM and L1 cache, offers faster access speeds. L1 cache has even faster access speeds but relatively smaller capacity. L0 cache usually refers to registers or local storage within the NPU, boasting extremely high access speeds; it's used to store data and instructions for currently executing threads or tasks. UB (Underlying Buffer) is an intermediate data buffer used for temporary storage before data is transferred from the L2 cache to the Vec (Vacuum Controller) compute unit. It helps balance data flow and computational load, improving overall computational efficiency. UB can also be referred to as the cache for the Vec compute unit, which retrieves data from it. It helps balance data flow and computational load, improving overall computational efficiency.

[0107] As an example, the GM can have a storage capacity of 64G to meet the needs of large-scale neural network models; the L2 cache can have a capacity of 192M to store a large amount of frequently used data and instructions, thereby reducing the number of times the GM is accessed and improving computational efficiency; the L1 cache can have a capacity of 512K to store recently accessed data and instructions, thereby further reducing access to the L2 cache and GM.

[0108] The Cube computing unit is a high-performance computing unit in the NPU, dedicated to performing tensor computations and matrix operations in neural networks. Through its highly parallel computing architecture, the Cube computing unit can efficiently handle large-scale neural network inference tasks. The Vec computing unit, another important computing unit in the NPU, focuses on performing vector computations. Vector computation is a common operation in neural networks, such as weight updates and activation function calculations. The Vec computing unit efficiently handles these tasks by optimizing vector computation paths and instruction sets.

[0109] In addition, Figure 4 above also illustrates the data paths in a multi-layer memory architecture. MTE3 refers to the data path from L1 cache / UB to GM / L2, denoted as MTE3_cube and MTE3_vec respectively; MTE2 refers to the data path from L2 cache to L1 / UB cache, denoted as MTE2_cube and MTE2_vec respectively; MTE1 refers to the data path from L1 cache to L0 cache; and Fixp refers to the data path from L0 cache to GM / L2.

[0110] As shown in Figures 3 and 4 above, the efficient computation of the core computing units (i.e., Tensor Cores, CUDA Cores, or Cube and Vec computing units) in both GPUs and NPUs relies on the layer-by-layer transfer of data from main memory (DRAM or GM). Especially in the NPU architecture, where the Cube and Vec computing units are designed separately, for the FA operator, this involves interactive computation between the two units. Besides the data transfer from GM to the computing unit, the main time-consuming part is the data loop transfer between the Cube and Vec computing units, which is often blocked in the fixp backhaul (transfer from high-speed memory L0 to low-speed memory L2 / HBM). Therefore, reducing precision can directly eliminate memory transfer blockage and significantly improve performance.

[0111] 6. Low-precision technology

[0112] Because the computational flow of FA involves complex multi-operator operations, such as matrix multiplication (matmul) and softmax, operator fusion techniques are generally used to significantly improve data locality and overall operator performance. Therefore, FA operators are also called fused operators. As shown in Table 1 above, the most significant technological advancement in FA from FA 1.0 to FA 3.0 is the improvement of operator performance through low-precision quantization and low-bit quantization. Furthermore, considering the data transmission within the aforementioned hardware architecture, reducing precision can decrease memory transfer congestion in the NPU. Therefore, to improve computational efficiency and performance, FA operators are often subjected to low-precision computation.

[0113] In this context, "low-precision" refers to directly adjusting the floating-point precision, i.e., using lower-precision floating-point numbers for calculations. For example, a traditional deep learning model might use fp32 for calculations, but low-precision techniques might attempt to use fp16 or even fp8. Since lower-precision floating-point numbers require less storage space, low-precision can reduce memory usage. Furthermore, modern hardware is specifically optimized for lower-precision floating-point calculations, thus improving computational performance.

[0114] It's important to note that low-bit quantization can be considered a special form of low-precision quantization, which further converts floating-point numbers into fixed-point integers for computation. For example, it uses 8-bit integers (int8) or 4-bit integers (int4) instead of fp16 for calculations. Since fixed-point integer calculations are generally simpler and faster than floating-point calculations, low-bit quantization not only further reduces memory usage but also improves computational efficiency.

[0115] As an example, Table 2 illustrates the low-precision calculation principle of the FA operator.

[0116] Table 2

[0117] It should be noted that the explanations of each calculation step and the meaning of the parameters in Table 2 can be found in the explanation of the Attention calculation process above, and will not be repeated here.

[0118] As shown in Table 2, in the case of reducing the precision to fp16, only the output of step 1 above is reduced from fp32 to a lower precision, such as fp16. Thus, after the fractional matrix S is calculated, only the low-precision variable (fp16) needs to be transmitted, reducing the amount of data transmitted by half compared to fp32. Furthermore, on some hardware (such as Tensor Cores or Cube computing units that support fp16 calculations), using low-precision data can accelerate computation. However, since the calculation of the fractional matrix S involves multiplication and addition operations, the intermediate and final results may be quite large, and using low precision (e.g., fp16) can easily lead to numerical overflow.

[0119] As shown in Table 2, when all fp16 values ​​are reduced to low precision, steps 1 to 4 above are all calculated using fp16. This further improves performance and reduces data transfer, achieving a performance improvement of over 30% compared to the original FA calculation process. However, when the value exceeds the maximum range that fp16 can represent, overflow occurs, causing the result to become positive infinity (INF) or result in not-a-number (NAN), which may lead to the interruption of the FA calculation process or the output of meaningless results. When the value is below the minimum range that fp16 can represent, underflow occurs, resulting in precision loss. Although underflow does not immediately cause computational crashes, the long-term cumulative precision loss may affect model performance.

[0120] Next, we will introduce and explain the representation range and overflow situation of data with different precision.

[0121] For quarter-precision (i.e., fp8) data, its numerical range can be represented by the following formula (1).

[0122] For fp8 data, when the value x is greater than a certain threshold (e.g., 448, which may vary depending on the specific implementation), it cannot be represented precisely and may overflow. Therefore, when the value x is greater than 448, it is represented as infinity (INF); when x is less than 448, a rounding function fl(x) is used to ensure that the value is within the range of fp8 representation.

[0123] It should be understood that fl(x) here refers to the rounding operation of floating-point numbers, but the specific implementation details may vary depending on different hardware and software platforms, and the embodiments of this application do not limit this.

[0124] For half-precision (i.e., fp16) data, its numerical range can be represented by the following formula (2).

[0125] For fp16 data, overflow may occur when the value x is greater than 65504 (again, this threshold may vary depending on the specific implementation). Similar to fp8, when x is greater than 65504, it is represented as infinity (INF); when x is less than 65504, the rounding function fl(x) is used.

[0126] For single-precision (i.e., fp32) data, its numerical range can be represented by the following formula (3).

[0127] For FP32 data, its representation range is very large, up to 3.4 × 10⁻⁶. 38 The value is within the range of x. When x is within this range, it can be represented precisely; when x exceeds this range, overflow may occur, but the handling may vary depending on the specific hardware and software platform. In some cases, out-of-range values ​​may be represented as infinity (INF) or not-a-number (NAN).

[0128] Based on this, Table 3 below gives the numerical representation range for three different floating-point precisions (fp8, fp16, fp32), as well as the smallest positive non-zero number they can represent, the range of numerical values ​​that can be accurately represented (i.e., the range of values ​​that will not be significantly changed by rounding errors), and the threshold for overflow.

[0129] Table 3

[0130] Based on Table 3 above, the smallest positive non-zero number that fp8 can represent is 6.25 × 10⁻⁶. -2 This means that numbers smaller than this will be rounded to zero or close to zero in fp8. The range of values ​​that fp8 can accurately represent is (0, 1.95 × 10⁻⁶). -3 Values ​​within this range will not change significantly due to rounding errors. The upper bound of this range is an approximation, and the actual range may vary depending on the specific implementation. When the value is greater than 448, fp8 will overflow, meaning it cannot accurately represent the value. It is usually represented as infinity (INF) or other forms of error handling are applied. The smallest positive non-zero number that fp16 can represent is 4.88 × 10⁻⁶. -4 This is much smaller than the smallest positive nonzero number of fp8, meaning that fp16 can represent even smaller numbers. In (0, 5.96 × 10⁻⁶), ... -8Within a certain range, fp16 can accurately represent numerical values, but it will overflow when the value exceeds 65504. The smallest positive non-zero number that fp32 can represent is 5.96 × 10⁻⁶. -8 This is the smallest of the three precisions, meaning that fp32 can represent very small numbers. In the range (0, 1.40 × 10⁻⁶), ... -45 Within a certain range, fp32 can accurately represent numerical values. The upper bound of this range is very close to zero, indicating that fp32 has high precision when representing very small numbers, while its precision is lower when the value is greater than 3.4 × 10⁻⁶. 38 At this time, fp32 will overflow.

[0131] As an example, suppose that when the FA operator calculates matrix S through matrix multiplication, the element values ​​of matrices Q and K are large, resulting in large intermediate and final results. If fp32 is used for calculation, these large values ​​can be correctly represented and processed. However, if fp16 is used, due to the limited representation range of fp16, these large values ​​may exceed the representation range of fp16, leading to overflow or underflow.

[0132] Overflow example: Suppose that an element in matrix Q and matrix K has a value of 10000, then their product is 100000000, which exceeds the maximum range that fp16 can represent (about 65536), so an overflow will occur, causing the result to become INF or NAN.

[0133] Example of underflow: Suppose that an element in the Q and K matrices is very small, close to the smallest range that fp16 can represent. Then their product may be lower than the smallest positive number that fp16 can represent, causing underflow. Although it will not immediately cause computational crashes, it will result in a loss of precision.

[0134] In summary, while low-precision computation can improve performance and efficiency, it is necessary to handle numerical overflow issues to ensure the stability and accuracy of the model.

[0135] 7. Overflow prevention technology

[0136] Currently, inference operator libraries for large models mainly reuse operator libraries from the training phase. This is because training and inference share many similarities in operations, such as matrix multiplication and convolution, so reusing training operator libraries can save development time and resources. However, since the inference phase does not involve backpropagation (i.e., gradient calculation is not required), the requirement for numerical precision is relatively low; while in the training phase, to ensure the accuracy of gradient calculation, fp32 or higher precision is typically used. Therefore, in the inference phase, the current operator libraries, which are mainly based on fp32 single precision, contain a large amount of precision redundancy.

[0137] Given the reduced precision requirements during the inference phase, the industry has begun exploring the use of low-precision operators (such as fp16 and fp8) for inference to reduce computational load and memory consumption, thereby improving inference speed and efficiency. Furthermore, inference is primarily geared towards online services, which have high requirements for latency, robustness, and user experience. Low precision can significantly reduce computational load and memory consumption, thus lowering latency and improving service response speed. Moreover, due to the significant performance advantages of low-precision inference operators and the technical challenges of overflow prevention design, it has become a core competitive area for major manufacturers in the AI ​​field. The successful application of low-precision inference operators will greatly enhance the out-of-the-box performance of operator libraries, meaning users can quickly obtain high-performance inference services without sacrificing too much precision, thereby accelerating the popularization and application of AI technology.

[0138] However, as explained earlier, while lower precision brings performance improvements, it also introduces new challenges, especially the issue of numerical overflow. Therefore, designing overflow prevention mechanisms to ensure the stability and accuracy of low-precision inference has become a significant technical challenge.

[0139] Overflow prevention techniques have a long history in general software design and are an important means of ensuring software stability and security. In the design of system function libraries, such as the standard C library (libc), the math library (libm), and low-level mathematical libraries such as the basic linear algebra subprograms (BLAS), overflow prevention techniques are considered to be related to system security.

[0140] Numerical scaling is a common design method for preventing overflow, especially dynamic scaling. Dynamic scaling prevents overflow by dynamically adjusting the range and precision of numerical values.

[0141] As an example, Table 4 below shows a case of dynamic scaling to prevent overflow for a general operator.

[0142] Table 4

[0143] It should be noted that the design of the basic function is the key in the design of the general operator to prevent overflow. Table 4 uses the modulus of complex numbers as an example for illustration.

[0144] As shown in Table 4 above, in complex modular arithmetic, direct calculation may lead to overflow when the absolute value of x or y is very large. To avoid this, dynamic scaling can be used, which involves first calculating the larger absolute value of x and then using it as a scaling factor, thereby reducing the risk of overflow. For equilibrium dynamic scaling, this method introduces scaling matrices R and S to scale the original matrix A to balance the size of its elements, thus avoiding overflow. This method is particularly useful in matrix operations because it ensures numerical stability during the computation process.

[0145] Recently, the industry has applied dynamic scaling technology to matrix calculations in traditional neural networks, achieving some success. This indicates that dynamic scaling technology has certain application potential in AI model inference. However, for large-scale model FA fusion operators based on the Transformer architecture, dynamic scaling technology fails to fully consider the inherent mathematical properties of these large-scale models. The calculation methods of these large models may involve complex matrix operations, attention mechanisms, etc., and the numerical characteristics of these operations differ from those in general software. Moreover, while dynamic scaling technology can adjust the range and precision of numerical values, it does not eliminate the generation of large numbers at the root. In large-scale model FA fusion operators based on the Transformer architecture, the generation of large numbers may be due to factors such as the scale of the input data and the complexity of the model structure. Furthermore, once dynamic scaling technology is adopted in large-scale model FA fusion operators based on the Transformer architecture, it may affect all subsequent calculation processes. This may lead to problems such as decreased computational accuracy and increased inference latency. Since dynamic scaling technology requires dynamic adjustment of numerical values, this may increase additional computational overhead. At the same time, due to the complexity of large models, this additional computational overhead may lead to increased inference latency.

[0146] In summary, while dynamic scaling technology has been widely used in general software and systems, it faces challenges in large-scale model FA fusion operators based on the Transformer architecture. This necessitates that the design of overflow prevention techniques fully consider the inherent mathematical properties of computational methods in specific domains and eliminate the generation of large numbers at their source to ensure the stability and accuracy of AI model inference.

[0147] In the field of large models, the trend in overflow prevention design methods is towards customized enhancement. Due to the complexity and accuracy requirements of large models, general overflow prevention methods may not meet their needs. Therefore, static scaling methods have been introduced for specific operator libraries for large models, such as the FA operator.

[0148] Static scaling introduces a fixed scaling factor during the computation process. Taking attention calculation as an example, the steps typically include: calculating the product of the Q matrix and the K matrix, and applying the softmax function to normalize the product to obtain the weight matrix. In the calculation of the weight matrix, a static scaling factor σ is introduced, typically set to... (Where d is the dimension of the K matrix). This prevents overflow caused by backward gradient explosion.

[0149] However, if the overflow occurs before the matrix multiplication is calculated, i.e. before the product of the Q matrix and the K matrix, then the static scaling method will fail and cannot prevent numerical overflow in large model inference FA.

[0150] Furthermore, current AI hardware typically includes a saturation mode. When the computational data exceeds the hardware's maximum representation range, the hardware automatically truncates the data to the maximum representation value (e.g., half-precision 65504). This mechanism avoids program crashes caused by non-numerical representations (NANs) due to infinity (INF), and has low overhead, ensuring basic program operation.

[0151] However, hardware saturation mode cannot guarantee the correctness of operators and large models. This is because truncated data may no longer satisfy the mathematical properties of the original calculation, leading to inaccurate results. Therefore, industry operator libraries typically do not enable hardware saturation mode to prevent overflow, but instead prefer to use more precise overflow prevention methods, such as dynamic scaling or static scaling.

[0152] In summary, both static and dynamic scaling belong to software scaling methods. Dynamic scaling is too general, failing to consider the characteristics of large-scale FA (Automatic Factorial) models, and involves extensive matrix splitting and merging, which complicates the coordination process of tensors and vector units in AI chips, increasing instruction overhead. Static scaling, on the other hand, has poor generalization and may fail to prevent overflow in large-scale model inference. Furthermore, hardware saturation mode, by forcibly truncating numerical values, fundamentally alters the calculation results, causing issues with the correctness of FA operators and large-scale model calculations.

[0153] Secondly, the relevant background of the embodiments of this application will be introduced.

[0154] To improve the overall computational performance of the network model, low-precision techniques can be used to reduce computational load and memory consumption, thereby increasing inference speed and efficiency. However, while using low-precision techniques, it is necessary to optimize for numerical overflow issues to ensure the stability and accuracy of low-precision inference.

[0155] In deep learning, the core operators of many network architectures, such as convolutional neural networks (CNNs) and multiple layer perceptrons (MLPs), can be transformed into matrix multiplication operations. GPUs, as the primary computing platform for training and inference of deep learning models, are particularly adept at performing large-scale matrix multiplication operations through their internal Tensor Cores, providing high computational throughput and low latency.

[0156] However, when performing large-scale matrix multiplications, numerical overflow can occur. Numerical overflow can lead to inaccurate calculation results and may even cause the program to crash. To avoid this, it is usually necessary to scale the input matrix to ensure that the values ​​during the calculation remain within a representable range.

[0157] Related technologies provide a dynamic scaling method that determines a scaling factor based on the numerical information of the input matrix, scales the input matrix, performs scaled matrix multiplication on a Tensor Core, and finally restores the original matrix multiplication result using the scaling factor. This method specifically includes the following steps:

[0158] (1) Determine the dynamic scaling factors a and b.

[0159] Based on the numerical information of the input matrices X and Y, and preset overflow constraints (such as maximum absolute value, maximum norm, etc.), two dynamic scaling factors, a and b, are determined. The purpose of these two scaling factors is to ensure that the scaled matrices X' and Y' do not overflow when performing matrix multiplication.

[0160] (2) Matrix scaling.

[0161] Using determined scaling factors a and b, the input matrices X and Y are scaled using the following formulas (4) and (5) to obtain the scaled matrices X′ and Y′. X′=aX (4) Y′=bY (5)

[0162] (3) Perform matrix multiplication.

[0163] Matrix multiplication of scaled matrices X′ and Y′ is performed on the Tensor Core of the GPU to obtain the scaled result Z′.

[0164] (4) Restore the original matrix multiplication result.

[0165] Using the following formula (6), the scaled result Z′ is converted back to the original matrix multiplication result Z by using the reciprocal of the product of scaling factors a and b.

[0166] The aforementioned technologies, by dynamically scaling the input matrix and utilizing the GPU's Tensor Cores to perform matrix multiplication, address the numerical overflow problem that may occur during the execution of core operators in traditional artificial neural networks, thereby improving the numerical stability and efficiency of computation. However, they also have at least one of the following shortcomings:

[0167] (1) Problems in the scaling factor determination process.

[0168] Determining the scaling factor in the aforementioned technologies involves steps such as calculating the global maximum absolute value, non-zero average value, and computational range of the entire matrix. Since tensor cores and other tensor computing units are better suited for handling large-scale matrix multiplication operations than these global statistical calculations, these calculations typically require separate solutions using vector computing units or even scalar computing units. This necessitates transferring matrix data from the tensor core to the vector computing unit. Therefore, the process of calculating the scaling factor may reduce the utilization rate of efficient computing units like tensor cores; furthermore, the need to transfer data between different computing units increases data transfer overhead. Because this overhead accounts for a significant proportion of the overall computation process, it leads to a decrease in overall efficiency.

[0169] Furthermore, because this computing method relies on specific hardware architectures (such as GPU Tensor Cores), it is difficult to extend to other types of AI neural network chip architectures, such as NPUs and tensor processing unit (TPU) chips.

[0170] (2) Issues related to application goals and scenarios.

[0171] The aforementioned technologies primarily target traditional artificial neural networks, such as CNNs and MLPs. These networks have relatively simple structures, mainly relying on single-operator operations like matrix multiplication. However, with the development of deep learning, the Transformer architecture, based on the Attention mechanism, has gradually become the core architecture of current large-scale models. This architecture incorporates complex attention and self-attention mechanisms, no longer solely dependent on matrix multiplication. Therefore, the overflow prevention solutions mentioned above are difficult to extend to such complex neural network architectures.

[0172] (3) Preventing overflow but not eliminating the source of overflow.

[0173] The aforementioned techniques prevent overflow during computation by dynamically scaling the input matrix. However, this method only postpones potential overflow issues to later stages of the computation. If subsequent operations require low-precision calculations, these operations can still lead to overflow. Therefore, these techniques do not fundamentally eliminate the source of potential overflow data.

[0174] (4) Failure to cover the large model architecture of Transformer.

[0175] When the aforementioned technologies were proposed, the industry had already developed the Transformer architecture, primarily based on the Attention mechanism, which is widely used in almost all large-scale models. However, the overflow prevention solutions of these technologies did not cover this architecture, making it difficult to leverage the inherent mathematical principles of the Transformer to optimize the computational process.

[0176] To address the need for overflow prevention in deep learning network models and to address at least one deficiency in the aforementioned related technologies, this application provides a method for determining model output. For network models including attention mechanisms, the technical solution of this application processes the key matrix before performing matrix operations to obtain an intermediate matrix. Then, using the query matrix, intermediate matrix, and value matrix, the attention mechanism module determines the output matrix. Since the average value of each column element in the intermediate matrix is ​​less than the average value of the corresponding column elements in the transpose of the key matrix (i.e., the values ​​of elements in the intermediate matrix are attenuated relative to the transpose of the key matrix), numerical overflow and computational instability can be prevented at the computational source.

[0177] Finally, the implementation environment of the embodiments of this application will be described.

[0178] Referring to Figure 5, the method for determining the model output structure provided in this application embodiment can be applied to the entire inference process of any network model. The entire inference process of a large model includes stages such as large model network loading, prompt engineering, full inference (prefill), incremental inference (decoding), and outputting the inference results.

[0179] The prefill phase of full inference is a crucial preprocessing step in the large model inference process, primarily aimed at preparing for subsequent incremental inference phases. In this phase, the model loads and processes user input data, along with system-attached templates, to generate necessary intermediate results or caches (such as key-value caches). Taking key-value cache generation as an example, in a Transformer architecture model, the result of the current token is typically only related to previous tokens. Therefore, in the prefill phase, the large model computes all user inputs and generates corresponding key-value caches. These caches are reused in subsequent incremental inference phases to reduce computational overhead.

[0180] As an example, in a text generation task, the model calculates and generates the key-value value corresponding to each token based on the text sequence input by the user, and stores these values ​​in a cache.

[0181] The decoding phase of incremental inference is the core part of the large model inference process, and its main task is to generate tokens for the output sequence one by one. In this phase, the model gradually constructs a complete output sequence based on the KV cache generated in the Prefill phase and the already generated tokens. The generation of each token is based on the previous token and the current context information. To reduce computation, the decoding phase reuses the KV cache generated in the Prefill phase. During the generation of each token, the model only calculates the KV value corresponding to the current token and merges it with the value in the cache. In this way, the model avoids repeatedly calculating tokens that have already been calculated, thereby improving inference speed.

[0182] As an example, in a text generation task, the model generates the next character step by step based on the already generated text and the currently input character, until a complete sentence or paragraph is generated. The decoding stage is an iterative generation process. In each iteration step, the model generates a new token and adds it to the output sequence. When a preset termination condition is met (such as generating text of a certain length, encountering a terminator, etc.), the model stops generating and returns the final output sequence to the user or system.

[0183] In summary, the Prefill stage of full inference and the Decoding stage of incremental inference each play important roles in the large model inference process. Through the collaboration of these two stages, the large model can efficiently generate output sequences that meet user needs.

[0184] Furthermore, large-scale model inference service scenarios, such as online dialogue and question answering, place extremely high demands on experiments. Therefore, the network models and processes for large-scale model inference are operator-based, thereby calling high-performance operator libraries for acceleration. Among these, the operators relevant to the technical solution of this application are mainly the prompt flash attention (PFA) operator and the incremental flash attention (IFA) operator, collectively referred to as FA operators in this embodiment.

[0185] In some embodiments, since the PFA operator in full inference in actual inference services generally takes much longer than the IFA operator, the technical solution of this application is mainly used to eliminate the potential numerical overflow of low-precision FA operators (such as fp16 and fp8).

[0186] It should be understood that the operator library shown in Figure 5 also includes: matrix multiplication (Matmul) operator, root mean square layer normalization (RMSNorm) operator, softmax operator, layer normalization (Layernorm) operator, and rotate position embedding (RoPE) operator. These operators are merely illustrative examples. In practical applications, the operator library used by large models may include many more operators, and this embodiment does not impose any limitations on this.

[0187] Based on the above-described large model inference process, the technical solution provided in this application embodiment is specifically applied to the large model inference software system of AI computing system. Its essence is program code deployed on the underlying hardware (such as GPU, NPU, TPU) of AI server.

[0188] Referring to Figure 6, the technical solution provided in this application can be integrated into the operator library of the platform software, and provide the function of Attention calculation to the full inference stage of the large model through a high-level application programming interface (API) or a low-level API.

[0189] As an example, the program code of the technical solution in this application runs on the AI ​​computing device of the server, such as the memory of NPU, GPU, or TPU.

[0190] Referring again to Figure 6, when using the technical solution provided in this application to provide Attention calculation, the main functions involved include: data loading, offset matrix construction, KV Cache data preprocessing, fraction matrix calculation, softmax calculation, KV Cache data recovery, and output matrix calculation. The calculation steps such as data loading, fraction matrix calculation, softmax calculation, and output matrix calculation can refer to the FA calculation process or the Attention calculation process, as their calculation procedures are similar and will not be described in detail in the following embodiments.

[0191] In implementing the technical solution provided in this application, the offset matrix construction is included in the KV Cache data preprocessing process. It is mainly used to process the transpose of the key matrix to obtain an intermediate matrix, which participates in the Attention calculation. KV Cache data recovery is mainly used in the output matrix determination process to correct the deviation caused by the introduction of the intermediate matrix in the FA calculation process, so as to obtain the correct output matrix. For details of the implementation process, please refer to the method embodiment shown in Figure 8 below.

[0192] Furthermore, this application also provides a computer device. This computer device can be a single terminal device or server, or other devices, clusters, data centers, etc., including hardware such as processors, used to execute the method for determining model output results provided in this application. When implementing the method for determining model output results provided in this application, the processor can include, but is not limited to, CPUs, GPUs, NPUs, TPUs, etc., and this application does not impose any limitations on this.

[0193] Please refer to Figure 7, which is a schematic diagram of a computer device according to an embodiment of this application. The computer device may be the device shown in Figure 5 or Figure 6, including underlying hardware, and carrying and running an AI computing system. The computer device includes at least one processor 701, a communication bus 702, a memory 703, and at least one communication interface 704.

[0194] The processor 701 can be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0195] The communication bus 702 is used to transmit information between the aforementioned components. The communication bus 702 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in Figure 7, but this does not mean that there is only one bus or one type of bus.

[0196] The memory 703 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile optical disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 703 may exist independently and be connected to the processor 701 via a communication bus 702. The memory 703 may also be integrated with the processor 701.

[0197] Communication interface 704 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 704 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0198] As an example, processor 701 may include one or more CPUs, such as CPU0 and CPU1 as shown in Figure 7.

[0199] As an example, a computer device may include multiple processors, such as processor 701 and processor 705 shown in Figure 7. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0200] In some embodiments, the computer device may further include output devices and input devices. The output device communicates with the processor 701 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 701 and can receive user input in various ways. For example, the input device may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0201] In some embodiments, memory 703 is used to store program code 710 for executing the scheme of this application, and processor 701 can execute program code 710 stored in memory 703. The program code 710 may include one or more software modules, and the computer device can implement the method for determining the model output result provided in the embodiment of FIG8 below through processor 701 and program code 710 in memory 703.

[0202] It should be understood that the application scenarios and computing device architectures described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of computing device architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0203] After introducing the terminology, application scenarios, and implementation environment involved in the embodiments of this application, the method for determining the model output results provided in the embodiments of this application will be explained in detail below.

[0204] Figure 8 is a flowchart of a method for determining model output results provided in an embodiment of this application. This method can be applied to the processor of the aforementioned computer device 700, which runs a network model that includes a mechanism module. Referring to Figure 8, the method includes the following steps.

[0205] Step 801: Based on the current input information of the network model, determine the query matrix, key matrix, and value matrix through the attention mechanism module.

[0206] In this application embodiment, the network model can be a large model or a small model, and this application embodiment does not limit its scale (e.g., the number of model parameters). Taking a large model as an example, large models typically use deep neural networks as their basic structure. These networks consist of multiple layers, each containing a certain number of neurons. Neurons are connected by weights, which are continuously adjusted during training to learn the feature representation of the input data. As the number of network layers increases, the model can learn more abstract and higher-level features, thereby improving performance. In the field of natural language processing, the Transformer model is a common large model structure. The core of the Transformer model is the encoder-decoder architecture, where both the encoder and decoder consist of multiple Transformer blocks. Each Transformer block contains two main modules: an attention mechanism and a feedforward neural network.

[0207] The attention mechanism module captures the dependencies between different words in a sequence. By calculating the similarity between query vectors, key vectors, and value vectors, the attention mechanism assigns a weight to each word in the current input, reflecting the degree of association between that word and other words. The feedforward neural network further extracts and fuses the information output by the attention mechanism. The feedforward neural network typically consists of two fully connected layers, extracting deeper semantic information through nonlinear transformations.

[0208] Taking the text generation task as an example, its task running process is roughly as follows: The input data (such as text, images, etc.) is converted into word embedding vectors (for text data) or corresponding feature vectors (for other types of data), and these vectors serve as the input to the encoder. Then, the input vectors propagate forward through multiple layers of the encoder. Each layer uses the self-attention mechanism and the feed-forward neural network to extract the feature information of the input data and generate corresponding vector representations, which represent the core features of the input data. The decoder gradually generates the tokens of the target sequence based on the output of the encoder and the generated sequence. Each newly generated token is inferred based on the previous output and the input sequence. The decoder also uses the self-attention mechanism and the feed-forward neural network to process this information and generate the probability distribution of the next token. By selecting the token with the highest probability as the next generated token, the decoder can gradually construct the complete target sequence.

[0209] As an example, taking machine translation as an example, assume that we want to translate the English sentence "The cat sat on the mat" into Chinese. First, the network model converts the English sentence into word embedding vectors and extracts its feature information through the encoder. Then, the decoder gradually generates the remaining Chinese tokens (such as "sat on the mat") based on the output of the encoder and the generated Chinese tokens (for example, "猫" as the first generated token). Finally, the decoder outputs the complete Chinese sentence as the translation result.

[0210] Taking the question-answering system as an example, its task running process is roughly as follows: The question-answering system needs to understand the user's question, which involves natural language processing of the question, including steps such as word segmentation, part-of-speech tagging, and syntactic analysis. Through these steps, the question-answering system can extract the key information of the question, such as the topic of the question, the content being asked, etc. Next, the question-answering system needs to retrieve information related to the question in the knowledge base or database. This can be achieved through methods such as keyword matching and semantic similarity calculation, and the retrieved information will serve as the basis for subsequent reasoning and answering. Finally, the question-answering system needs to reason based on the retrieved information and the user's question and generate a reasonable answer. This also involves the use of an encoder and a decoder. The encoder is used to extract the feature representations of the question and the retrieved information, while the decoder is used to generate the answer based on these feature representations.

[0211] As an example, assume that the user asks a question: "Is Beijing the capital of China?" The question-answering system first understands the key information of the question: "Beijing" and "the capital of China". Then, it retrieves information related to these two keywords in the knowledge base. Finally, it reasons based on the retrieved information and the semantic relationship of the question and generates an answer: "Yes, Beijing is the capital of China".

[0212] Based on the above description, the current input information of the network model in step 801 can be the input information of the first network layer in the network model, i.e., the input sequence; when the network model includes multiple sequentially connected network layers, and each network layer includes an attention mechanism, the current input information can also be the input information of any intermediate network layer. This application embodiment does not impose any limitations on this.

[0213] In one possible implementation, step 801 can be implemented as follows: based on the current input information of the network model, the current input information is multiplied by the weight matrix of the corresponding network layer to obtain the query (Q) matrix, key (K) matrix, and value (V) matrix. Each row in the query matrix, key matrix, and value matrix corresponds to the query vector (also called the Q vector), key vector (also called the K vector), and value vector (also called the V vector) of one element in the current input information.

[0214] Referring to the relevant explanations in Figures 1 and 2 above, it should be understood that when the input sequence lengths are S1 and S2 and the feature dimension is d, the sequence length of the query matrix (i.e., the direction of the matrix columns) is S1, and the sequence lengths of the key matrix and value matrix are S2; each query vector, key vector, and value vector includes d elements.

[0215] Step 802: Process the transpose of the key matrix to obtain an intermediate matrix. The intermediate matrix has the same number of rows and columns as the transpose matrix, and the average value of each column element in the intermediate matrix is ​​less than the average value of the corresponding column elements in the transpose matrix.

[0216] It should be understood that for a key matrix, its transpose is obtained by swapping the rows and columns of the key matrix. In other words, the row elements of the transpose matrix are the column elements of the key matrix, and the column elements of the transpose matrix are the row elements of the key matrix. Transposing does not change the values ​​of the elements in the matrix.

[0217] In the first possible implementation, the above step 802 can be implemented as follows: obtain the offset parameter; construct an offset matrix based on the offset parameter and the number of columns of the transpose matrix, wherein each element in the offset matrix is ​​less than 1, the number of columns of the offset matrix is ​​the same as that of the transpose matrix, and the number of rows and columns of the offset matrix are equal; perform matrix multiplication on the transpose matrix and the offset matrix to obtain the intermediate matrix.

[0218] In this matrix, the number of rows and columns of the offset matrix are equal, meaning the offset matrix is ​​a square matrix. Based on this, a matrix multiplication operation is performed between the offset matrix and the transpose matrix to obtain an intermediate matrix. The number of rows in the intermediate matrix is ​​equal to the number of rows in the transpose matrix, and the number of columns in the intermediate matrix is ​​equal to the number of columns in the offset matrix.

[0219] It should be noted that the offset parameter can be a hyperparameter preset by the network model, or it can be a parameter determined based on the current input information. This application embodiment does not impose any restrictions on this.

[0220] In one possible implementation, the process of obtaining the offset parameter can be as follows: obtain the data type of any matrix among the query matrix, key matrix, and value matrix, where the data type indicates the floating-point precision of the data; and determine the offset parameter based on the data type from the correspondence between the data type and the offset parameter.

[0221] Specifically, based on the input data of the FA operator during network model inference—namely, the data types of the query matrix and key-value cache sequence—the optimal precision hyperparameters can be configured to ensure minimal error under low-precision calculations (e.g., fp16, fp8). As an example, the correspondence between data types and offset parameters obtained using optimal precision analysis is shown in Table 5 below.

[0222] Table 5

[0223] It should be noted that the data type of the input matrix (i.e., query matrix, key matrix, value matrix) at compile time is given through configuration files, such as JSON (a commonly used data exchange format for storing and transmitting configuration information). Therefore, the offset parameter α is a constant at compile time and resides in HBM memory.

[0224] In one possible implementation, with the offset parameter being α and the number of columns or rows of the offset matrix being s², the main diagonal elements of the constructed offset matrix are: The elements not on the main diagonal are

[0225] As an example, suppose the number of rows of the key matrix is ​​s2, then the number of columns of the corresponding transpose matrix is ​​s2. In this case, the offset matrix M is a square matrix of s2×s2, which can be expressed by the following formula (7).

[0226] It should be noted that, due to the main diagonal elements in the offset matrix Since these are positive numbers, during matrix multiplication between the offset matrix and its transpose, these diagonal elements are multiplied by the corresponding elements in the transpose matrix, thus preserving some of the values ​​of these elements in the transpose matrix. However, this effect is localized, limited to the positions in the transpose matrix corresponding to the diagonal elements of the offset matrix. This is because the non-diagonal elements of the offset matrix... The negative values ​​of these off-diagonal elements have a "decaying" effect on the elements of the transpose matrix during matrix multiplication. Specifically, when an element of the transpose matrix is ​​multiplied by the off-diagonal elements of the offset matrix, the result is negative, and the absolute value of this negative number depends on the ratio of α and s². Furthermore, this negative product is added to other products (including products with the off-diagonal elements of the offset matrix), potentially reducing the value of the corresponding element in the transpose matrix.

[0227] In summary, since all elements in the offset matrix are less than 1, and the main diagonal elements are... The elements not on the main diagonal are Therefore, when performing matrix multiplication between the offset matrix and the transpose matrix, the diagonal elements in the offset matrix will retain the values ​​of the corresponding elements in the transpose matrix to a certain extent, while the off-diagonal elements will have a significant attenuation effect on the elements in the transpose matrix. This effect will greatly reduce the values ​​of the elements in the transpose matrix during matrix multiplication, thereby preventing numerical overflow and computational instability at the source of attention computation.

[0228] In a second possible implementation, the transpose matrix comprises multiple transpose basic blocks, each containing elements with multiple rows and columns, and all elements in the multiple transpose basic blocks have the same number of rows and columns. Based on this, step 802 can be implemented as follows: obtaining offset parameters; constructing an offset matrix based on the offset parameters and the number of columns of the elements in the transpose basic blocks, where all elements in the offset matrix are less than 1, and the offset matrix has the same number of columns as the elements in the transpose basic blocks, and the number of rows and columns of the offset matrix are equal; performing matrix multiplication operations between the multiple transpose basic blocks and the offset matrix to obtain multiple intermediate basic blocks; and generating an intermediate matrix based on the multiple intermediate basic blocks.

[0229] Unlike the first implementation mentioned above, as shown in Figure 2 above, in order to accelerate computation and reduce memory usage, the FA operator can divide the input matrix (i.e., query matrix, key matrix, and value matrix) into blocks when calculating attention, and perform matrix operations on the basis of basic blocks.

[0230] As an example, referring to Figure 9, assume that after partitioning the transpose of the key matrix, eight transpose basic blocks are obtained, all of which are of the same size. When constructing the offset matrix, the number of rows and columns of this offset matrix are equal to the number of columns of each transpose basic block. Then, matrix multiplication is performed between each of the eight transpose basic blocks and the offset matrix to obtain eight corresponding intermediate basic blocks. These eight intermediate basic blocks are then concatenated according to the partitioning method described above to obtain the intermediate matrix.

[0231] The values ​​of the offset parameters, the element values ​​in the offset matrix, and the beneficial effects of generating the intermediate matrix can be found in the first implementation method described above. The construction logic of the two methods is the same, and will not be elaborated upon here.

[0232] Step 803: Based on the query matrix, intermediate matrix, and value matrix, determine the output matrix through the attention mechanism module.

[0233] When determining the output matrix using an attention mechanism, a standard attention mechanism can be used, as shown in Figure 1; alternatively, an optimized attention algorithm (FA) calculation process can be employed, as shown in Figure 2. This application does not impose any limitations on this approach.

[0234] It's important to note that with standard attention mechanisms, the computation process involves matrix multiplication of the entire matrix followed by softmax normalization. Because softmax is translation-invariant, even if the intermediate matrix is ​​obtained by transposing the key matrix, the correctness of the final output matrix remains unaffected. However, with the FA (Automatic Focusing) computation process, matrix multiplication is performed on a block-by-block basis, and softmax normalization is applied. The average value of each block may differ, potentially leading to biases in the processing results and consequently, biases in the output matrix. Therefore, in block-based processing, corrections to the output blocks are necessary to obtain the correct output matrix.

[0235] Next, taking the FA operator as an example, the process of determining the output matrix using the technical solution of this application will be explained. Specifically, during the attention calculation process of the FA operator, the input matrix can be divided into blocks, resulting in multiple basic blocks corresponding to each matrix. Based on this, the query matrix includes multiple query basic blocks, the intermediate matrix includes multiple intermediate basic blocks, and the value matrix includes multiple value basic blocks.

[0236] Referring again to Figure 2, with the sequence length of the query matrix (i.e., the direction of the matrix columns) being S1, the sequence lengths of the key matrix and value matrix being S2, and the feature dimension being d, the number of multiple query basic blocks is S1 / s1, and the number of multiple key basic blocks and multiple value basic blocks is S2 / s2. The size of a single query basic block is s1×d, and the size of a single key basic block and a single value basic block is s2×d.

[0237] In one possible implementation, referring to Figure 10, the implementation process of step 803 may include the following steps 8031-8035.

[0238] Step 8031: Perform matrix multiplication on multiple query basic blocks and multiple intermediate basic blocks to obtain multiple fractional basic blocks.

[0239] Referring to Figure 11, for each of the multiple query basic blocks, a matrix multiplication operation is performed between that query basic block and each of the multiple intermediate basic blocks to obtain a row of fractional basic blocks corresponding to that query basic block. Each row of fractional basic blocks includes multiple fractional basic blocks, and the number of multiple fractional basic blocks is equal to the number of multiple intermediate basic blocks. This process is repeated to obtain all the fractional basic blocks obtained after performing matrix multiplication operations between multiple query basic blocks and multiple intermediate basic blocks.

[0240] It should be understood that the number of fractional basic blocks obtained in step 8031 ​​above is the product of the number of query basic blocks and the number of intermediate basic blocks. For example, if the number of query basic blocks is 8 and the number of intermediate basic blocks is 8, then after the matrix multiplication operation in step 8031 ​​above, 64 fractional basic blocks will be obtained.

[0241] The multiple fractional basic blocks are distributed in M ​​rows and N columns, where M and N are both integers greater than or equal to 1. M represents the number of query basic blocks, and N represents the number of intermediate basic blocks. Therefore, for each fractional basic block, the number of rows in the fractional basic block is equal to the number of rows in the corresponding query basic block; the number of columns in the fractional basic block is equal to the number of columns in the corresponding intermediate basic block.

[0242] As an example, if the size of the query basic block is s1×d and the size of the intermediate basic block is s2×d, then the size of the fractional basic block is s1×s2.

[0243] Step 8032: Determine multiple weight basic blocks based on multiple score basic blocks.

[0244] It should be noted that the number and size of the multiple weight basic blocks are the same as those of the multiple score basic blocks. The only difference is that the values ​​of the elements in each score basic block and its corresponding weight basic block are different. Therefore, the multiple weight basic blocks are also distributed in M ​​rows and N columns, where M and N are both integers greater than or equal to 1. M is the number of multiple query basic blocks, and N is the number of multiple intermediate basic blocks. Furthermore, the number of rows of elements included in the multiple weight basic blocks is the same.

[0245] In one possible implementation, the softmax function is applied to the elements of each row in each fractional block to convert them into probability values, thereby transforming the fractional block into a weighted block.

[0246] The softmax function is a function that transforms a vector of real numbers into a probability distribution. It converts any real number into a value between 0 and 1, and the sum of these values ​​is 1. In the score block, the softmax function normalizes the values ​​of each element (i.e., the attention score) in each row of the score block to obtain the weight value of each element relative to the other elements. Therefore, the sum of the values ​​of each element in each row of the weight block is 1, and the value of each element is between 0 and 1.

[0247] As an example, the softmax function can be represented by the following formula (8).

[0248] In this formula, qi is the i-th element in a row of elements in the fractional basic block, k is the total number of elements in that row, and the denominator is the sum of the exponents of all elements in that row. This sum of exponents serves as a normalization constant to ensure that the sum of the element values ​​in that row after normalization is 1.

[0249] As another example, the softmax function can also be expressed by the following formula (9), and the two calculation methods are equivalent.

[0250] In this formula, qi is the i-th element in a row of elements in the fractional basic block, k is the total number of elements in that row, and max(q) is the maximum value of the elements in that row.

[0251] In addition, the implementation process of step 8032 above can also refer to the Attention calculation process shown in Figure 1 or the FA operator calculation process shown in Figure 2. The calculation logic is similar, so it will not be described again here.

[0252] Step 8033: Determine multiple output basic blocks based on multiple weight basic blocks and multiple value basic blocks.

[0253] Since multiple weight basic blocks are also distributed in M ​​rows and N columns, and the number of rows of elements in multiple weight basic blocks is the same; and multiple value basic blocks are distributed in N rows, and the number of columns of elements in multiple value basic blocks is the same, then performing matrix multiplication between the weight basic block in the i-th row and the value basic block in the i-th row will yield the output basic block corresponding to the i-th row.

[0254] In this configuration, multiple output basic blocks contain elements with the same number of rows, and the weight basic blocks and output basic blocks also contain elements with the same number of rows; multiple output basic blocks contain elements with the same number of columns, and the value basic blocks and output basic blocks also contain elements with the same number of columns. In other words, the number of rows in the output basic block of row i is equal to the number of rows in the weight basic block of row i, and the number of columns in the output basic block of row i is equal to the number of columns in the value basic block of row i.

[0255] In addition, the implementation process of step 8033 above can also refer to the Attention calculation process shown in Figure 1 or the FA operator calculation process shown in Figure 2. The calculation logic is similar, so it will not be described again here.

[0256] Step 8034: Perform offset correction on multiple output basic blocks to obtain corrected multiple output basic blocks.

[0257] It should be noted that since the output basic blocks are determined based on the weight basic blocks and the value basic blocks, and there are deviations in the process of calculating the fractional basic blocks, the values ​​of the elements in the weight basic blocks will also be deviated after the fractional basic blocks are converted into weight basic blocks. Therefore, when performing offset correction on the output basic blocks, it is necessary to use the relevant information of the fractional basic blocks and the weight basic blocks.

[0258] The relevant information for the fractional basic blocks and weighted basic blocks includes: the maximum value of each row element in the fractional basic block, the row sum of each row element in the weighted basic block, the cumulative row maximum value of multiple fractional basic blocks, and the cumulative row sum of multiple weighted basic blocks.

[0259] In one possible implementation, the relevant information of the fractional basic block and the weight basic block can be iteratively updated during the calculation of the weight basic block, so that while determining the weight basic block of the k-th row, the global row sum corresponding to the multiple row elements of the weight basic block of the k-th row can also be determined, as well as the maximum value corresponding to the multiple row elements of the fractional basic block of the k-th row.

[0260] The global row sum refers to the sum of a row of elements in the k-th row of the weighted basic block, where k is greater than or equal to 1 and less than or equal to M.

[0261] Taking a single basic block as an example, as shown in Figure 12, for a fractional basic block S1 among multiple fractional basic blocks, in the process of using the softmax function to calculate the weighted basic block P1 corresponding to the fractional basic block S1, the row sum of each row element in the weighted basic block P1 is calculated, and the maximum value of each row element in the fractional basic block S1 is calculated.

[0262] It should be understood that, assuming the fractional basic block S1 is a 3×3 matrix with 9 elements, then the weight matrix P1 is also a 3×3 matrix, the row sum (rowsum) of each row element in the weight basic block P1 is a 3×1 matrix, and the maximum value (rowmax) of each row element in the fractional basic block S1 is also a 3×1 matrix.

[0263] In another possible implementation, after determining multiple fractional basic blocks, the maximum values ​​corresponding to the multiple row elements of the fractional basic block in the k-th row can be calculated; after determining multiple weighted basic blocks, the global row sums corresponding to the multiple row elements of the weighted basic block in the k-th row can be calculated.

[0264] In other words, the embodiments of this application do not limit the timing of determining the relevant information of the fractional basic block and the weight basic block, and can be selected according to the needs of the actual application.

[0265] In some embodiments, the process of offset correction of multiple output basic blocks to obtain corrected multiple output basic blocks may include the following steps 1-3.

[0266] Step 1: Determine the global row sum of the weight basic block in the k-th row of multiple weight basic blocks. The global row sum is the sum of the elements in one row of the weight basic block in the k-th row, where k is greater than or equal to 1 and less than or equal to M.

[0267] Since each weighted basic block is also a small matrix containing multiple rows of elements, calculating the global row sum of the weighted basic block in the k-th row is equivalent to calculating the row sum corresponding to each row of elements in these k weighted basic blocks.

[0268] In one possible implementation, the process of determining the sum of multiple global rows of the weighted basic block in the k-th row of multiple weighted basic blocks may include the following steps 1.1-1.4.

[0269] Step 1.1: Let j = 1, determine the correction parameter for the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th row, and the correction parameter for the row sum of the i-th row elements of the j-th weight basic block. The row sum is the sum of the i-th row elements.

[0270] As explained earlier, there is a one-to-one correspondence between multiple weight basic blocks and multiple fraction basic blocks. The number of rows of elements in each fraction basic block is the same, and the number of rows of elements in each fraction basic block is also the same as that in the weight basic blocks. Furthermore, the number of rows of elements in each output basic block is the same, and the number of rows of elements in each weight basic block is also the same as that in the output basic blocks. Therefore, i is greater than or equal to 1, and less than or equal to the number of rows of elements in each output basic block (or, in other words, the weight basic block or fraction basic block).

[0271] In one possible implementation, the process of implementing step 1.1 above may include the following steps (1)-(4).

[0272] (1) Obtain the correction parameter of the maximum value of the i-th row element of the first j-1 fractional basic blocks in the k-th row, and the correction parameter of the maximum value of the i-th row element of the j-th fractional basic block.

[0273] In one possible implementation, step (1) is implemented as follows: obtain the average value of the i-th row element of the first j-1 fractional basic blocks, the average value of the i-th row element of the j-th fractional basic block, and the average value of the i-th row element of the first j-1 fractional basic blocks; obtain the offset parameter; based on the average value of the i-th row element of the first j-1 fractional basic blocks, the average value of the i-th row element of the j-th fractional basic block, the average value of the i-th row element of the first j-1 fractional basic blocks, and the offset parameter, determine the correction parameter for the maximum value of the i-th row element of the first j-1 fractional basic blocks and the correction parameter for the maximum value of the i-th row element of the j-th fractional basic block.

[0274] It should be noted that the offset parameter is the same as α mentioned earlier. The process of obtaining the offset parameter can be found in the relevant description in step 802 above, and will not be repeated here.

[0275] In one possible implementation, the process of calculating the average of the i-th row elements of the first j fractional basic blocks can be as follows: based on the average of the i-th row elements of the first j-1 fractional basic blocks and the average of the i-th row elements of the j-th fractional basic block, determine the average of the i-th row elements of the first j fractional basic blocks.

[0276] As an example, the average value of the i-th row of the first j fractional basic blocks can be determined by the following formula (10).

[0277] in, Let be the average of the elements in the i-th row of the first j fractional basic blocks. Let be the average of the elements in the i-th row of the first j-1 fractional basic blocks. It is the average value of the elements in the i-th row of the j-th fractional basic block.

[0278] As an example, in conjunction with the offset parameter, the correction parameter for the maximum value of the i-th row element of the first j-1 fractional basic blocks can be determined by the following formula (11).

[0279] Where α is the offset parameter. Let be the average of the elements in the i-th row of the first j-1 fractional basic blocks. Let Δm be the average of the elements in the i-th row of the first j fractional basic blocks. j-1 It is a correction parameter for the maximum value of the i-th row element of the first j-1 fractional basic blocks.

[0280] As an example, in conjunction with the offset parameter, the correction parameter for the maximum value of the i-th row element of the j-th fractional basic block can be determined by the following formula (12).

[0281] Where α is the offset parameter. Let be the average of the elements in the i-th row of the j-th fractional basic block. Let Δm be the average of the elements in the i-th row of the first j fractional basic blocks. j The correction parameter is the maximum value of the element in the i-th row of the j-th fractional basic block.

[0282] (2) Get the maximum value of the i-th row of the first j-1 fractional basic blocks, the maximum value of the i-th row of the j-th fractional basic block, and the maximum value of the i-th row of the first j fractional basic blocks.

[0283] In one possible implementation, the process of obtaining the maximum value of the i-th row element of the first j fractional basic blocks can be as follows: Based on the correction parameter of the maximum value of the i-th row element of the first j-1 fractional basic blocks, the maximum value of the i-th row element of the first j-1 fractional basic blocks is corrected to obtain the corrected maximum value of the i-th row element of the first j-1 fractional basic blocks; Based on the correction parameter of the maximum value of the i-th row element of the j-th fractional basic block, the maximum value of the i-th row element of the j-th fractional basic block is corrected to obtain the corrected maximum value of the i-th row element of the j-th fractional basic block; From the corrected maximum value of the i-th row element of the first j-1 fractional basic blocks and the corrected maximum value of the i-th row element of the j-th fractional basic block, the maximum value of the i-th row element of the first j fractional basic blocks is determined.

[0284] As an example, based on the correction parameter of the maximum value of the i-th row element of the first j-1 fractional basic blocks determined in step (1) above, the corrected maximum value of the i-th row element of the first j-1 fractional basic blocks can be determined by the following formula (13). j-1 =m j-1 +Δm j-1 (13)

[0285] Where, m j-1 Δm represents the maximum value of the i-th row element in the first j-1 fractional basic blocks. j-1 The correction parameter, m″, is the maximum value of the element in the i-th row of the first j-1 fractional basic blocks. j-1 It is the maximum value of the i-th row element of the first j-1 fractional basic blocks after correction.

[0286] As an example, based on the correction parameter of the maximum value of the i-th row element of the j-th fractional basic block determined in step (1) above, the corrected maximum value of the i-th row element of the j-th fractional basic block can be determined by the following formula (14). j =m′ j +Δm j(14)

[0287] Where, m′ j Δm represents the maximum value of the element in the i-th row of the j-th fractional basic block. j The correction parameter, m″, is the maximum value of the element in the i-th row of the j-th fractional basic block. j It is the maximum value of the i-th row element of the j-th fractional basic block after correction.

[0288] Furthermore, based on the maximum value of the i-th row element of the first j-1 fractional basic blocks after correction as determined by the above formula (13), and the maximum value of the i-th row element of the j-th fractional basic block after correction as determined by the above formula (14), the maximum value of the i-th row element of the first j fractional basic blocks can be determined by the following formula (15). j =max(m″ j-1 ,m″ j (15)

[0289] Where, m″ j-1 The maximum value after correction for the i-th row element of the first j-1 fractional basic blocks, m″ j m is the maximum value of the i-th row element of the j-th fractional basic block after correction. j The max() function is used to determine the maximum value in a set of values, where the maximum value is the maximum value in the i-th row of the first j fractional basic blocks.

[0290] (3) Based on the correction parameters of the maximum value of the i-th row element of the first j-1 fractional basic blocks, the maximum value of the i-th row element of the first j fractional basic blocks, and the maximum value of the i-th row element of the first j-1 fractional basic blocks, determine the correction parameters of the row sum of the i-th row elements of the first j-1 weighted basic blocks.

[0291] As an example, the correction parameter for the row sum of the i-th row elements of the first j-1 weighted basic blocks can be determined using the following formula (16): Δm″ j-1 =m j-1 -m j +Δm j-1 (16)

[0292] Where, m j-1 Let m be the maximum value of the i-th row element of the first j-1 fractional basic blocks. j Let Δm be the maximum value of the i-th row element of the first j fractional basic blocks. j-1 Δm″ is the correction parameter for the maximum value of the i-th row element of the first j-1 fractional basic blocks. j-1 It is a correction parameter for the row sum of the i-th row elements of the first j-1 weighted basic blocks.

[0293] (4) Based on the maximum value of the i-th row element of the j-th fractional basic block, the maximum value of the i-th row element of the first j fractional basic blocks, and the correction parameter of the maximum value of the i-th row element of the j-th fractional basic block, determine the correction parameter of the row sum of the i-th row element of the j-th weighted basic block.

[0294] As an example, the correction parameter for the row sum of the i-th row elements of the first j-1 weighted basic blocks can be determined using the following formula (17): Δm″ j =m′ j -m j +Δm j (17)

[0295] Where, m′ j Let m be the maximum value of the i-th row element of the j-th fractional basic block. j Let Δm be the maximum value of the i-th row element of the first j fractional basic blocks. j Δm″ is the correction parameter for the maximum value of the element in the i-th row of the j-th fractional basic block. j The correction parameter is the row sum of the i-th row element of the j-th weighted basic block.

[0296] It should be noted that, in this embodiment, the correction parameters for the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th row of the weight basic block, and the correction parameters for the row sum of the i-th row elements of the j-th weight basic block, determined in step 1.1 above, will be used in step 1.3 below to correct the row sum of the i-th row elements of the first j weight basic blocks in the k-th row of the weight basic block. Furthermore, they will also be used in step 2 below to correct the i-th row elements of the k-th row of the output basic block in the plurality of output basic blocks.

[0297] In summary, referring to Figure 13, this application embodiment provides an implementation process for determining the correction parameters of the row sum. Specifically, when determining the correction parameters of the row sum of each row element of the weight basic block of the k-th row based on the fraction basic block of the k-th row, the specific calculation methods involved are detailed in the above formulas (10)-(17), and will not be repeated here.

[0298] Step 1.2: Obtain the row sum of the i-th row of the first j-1 weighted basic blocks, and the row sum of the i-th row of the j-th weighted basic block.

[0299] It should be understood that the sum of the elements in the i-th row of the first j-1 weighted basic blocks is obtained by summing the sum of the elements in the i-th row of the first j-2 weighted basic blocks and the sum of the elements in the i-th row of the (j-1)-th weighted basic block. The sum of the elements in the i-th row of the j-th weighted basic block is obtained by summing the elements in the i-th row for the j-th weighted basic block.

[0300] Step 1.3: Based on the correction parameters of the row sum of the i-th row elements of the first j-1 weight basic blocks, the row sum of the i-th row elements of the first j-1 weight basic blocks, the correction parameters of the row sum of the i-th row elements of the j-th weight basic block, and the row sum of the i-th row elements of the j-th weight basic block, determine the corrected row sum of the i-th row elements of the first j weight basic blocks in the k-th weight basic block.

[0301] As an example, the row sum of the elements in the i-th row of the first j weighted basic blocks in the k-th row can be determined by the following formula (18). j =exp(Δm″ j-1 )l j-1 +exp(Δm″ j )l′ j (18)

[0302] Where exp() represents the exponential function, Δm″ j-1 l is a correction parameter for the row sum of the i-th row elements of the first j-1 weighted basic blocks. j-1 Δm″ is the row sum of the i-th row elements of the first j-1 weighted basic blocks. j l′ is the correction parameter for the row sum of the i-th row element of the j-th weighted basic block. j Let l be the row sum of the i-th row element of the j-th weighted basic block. j The sum of rows is the adjusted sum of the elements in the i-th row of the first j weighted basic blocks.

[0303] It should be understood that when j equals N, l j This is the global row sum of the elements in the i-th row of the weighted basic block in the k-th row.

[0304] Step 1.4: If j = N, then determine the global row sum of the i-th row elements of the first j weight basic blocks after correction. If j < N, then let j = j + 1, and return the steps for determining the correction parameters of the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th weight basic block, and the correction parameters of the row sum of the i-th row elements of the j-th weight basic block.

[0305] That is, by repeatedly executing steps 1.1-1.3 above, the row sums of all elements in each row of all weighted basic blocks in row k are corrected in turn, and finally the global row sums of all elements in each row of weighted basic blocks in row k are obtained.

[0306] It should be noted that since the number of rows in the weighted basic block is the same as the number of rows in the output basic block, the global row sum of the elements in each row of the weighted basic block in the kth row will be used in the subsequent step 3 to globally correct the elements in the output basic block in the kth row, and finally obtain the corrected output basic block in the kth row.

[0307] Step 2: Correct the i-th row element of the k-th row of the output basic block in multiple output basic blocks to obtain the corrected i-th row element of the k-th row of the output basic block, where i is greater than or equal to 1 and less than or equal to the number of rows of elements included in the output basic block.

[0308] In this context, modifying the element in the i-th row essentially means modifying the value of the element in the i-th row.

[0309] In one possible implementation, the process of modifying the i-th row element of the k-th row of the output basic block in multiple output basic blocks may include the following steps 2.1-2.4.

[0310] Step 2.1: Let j = 1, determine the correction parameter for the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th row, and the correction parameter for the row sum of the i-th row elements of the j-th weight basic block. The row sum is the sum of the i-th row elements.

[0311] The process of determining the correction parameters for the row sum of the i-th row elements of the first j-1 weighted basic blocks and the correction parameters for the row sum of the i-th row elements of the j-th weighted basic block can be referred to in step 1.1 above. The determination methods for the two are the same, so they will not be repeated here.

[0312] Step 2.2: Obtain the O values ​​from the output basic block in the k-th row. ig The first and second values ​​of the element, O ig An element refers to the element located in the i-th row and g-th column of the basic block of output in the k-th row.

[0313] The first value is determined based on the i-th row element of the first j-1 weight basic blocks and the g-th column element of the k-th value basic block. The second value is determined based on the i-th row element of the j-th weight basic block and the g-th column element of the k-th value basic block. g is greater than or equal to 1 and less than or equal to the number of columns of elements included in the value basic block.

[0314] Step 2.3: Based on the first value, the second value, the correction parameter of the row sum of the i-th row elements of the first j-1 weighted basic blocks, and the correction parameter of the row sum of the i-th row elements of the j-th weighted basic block, determine the O in the output basic block of the k-th row. ig The third value of an element.

[0315] As an example, the O values ​​in the output basic block of the k-th row can be determined using the following formula (19). ig The third value of the element. O″′ ig =exp(Δm″ j-1 )O′ ig +exp(Δm″ j )O″ ig (19)

[0316] Where exp() represents the exponential function, Δm″ j-1 O′ is the correction parameter for the row sum of the i-th row elements of the first j-1 weighted basic blocks. ig For the output basic block in the k-th row, O ig The first value of the element, Δm″ j O″ is the correction parameter for the row sum of the i-th row element of the j-th weighted basic block. ig For the output basic block in the k-th row, O ig The second value of the element, O″′ ig For the output basic block in the k-th row, O ig The third value of an element.

[0317] Step 2.4: If j = N, then change the O values ​​in the output basic block of the k-th row. ig The third value of the element is determined as the value of the corrected i-th row element of the output basic block in the k-th row; if j < N, then let j = j + 1, and return the steps for determining the correction parameters of the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th row, and the correction parameters of the row sum of the i-th row elements of the j-th weight basic block.

[0318] That is, by repeatedly executing steps 2.1-2.3 above, the elements of each row of the output basic block in the kth row are modified in turn, and finally the modified values ​​of each row of the output basic block in the kth row are obtained.

[0319] Step 3: Divide each element of the corrected i-th row of the output basic block of the k-th row by the corresponding global row sum in the multiple global row sums to obtain the corrected output basic block of the k-th row.

[0320] Here, the modified element O of the i-th row of the output basic block in the k-th row is taken as... ig For example, O igThe third value of an element is divided by the global row sum of the elements in the i-th row of the weighted basic block in the k-th row to obtain the O. ig The final value of the element. Similarly, after determining the final value of the element in the i-th row of the corrected output basic block in the k-th row, the corrected basic block in the k-th row can be obtained.

[0321] As an example, the O values ​​in the basic block of the output in the k-th row can be determined using the following formula (20). ig The final value of the element.

[0322] Among them, O″′ ig For the output basic block in the k-th row, O ig The third value of an element, l j The global row sum of the i-th row element of the weighted basic block in row k, O ig For the output basic block in the k-th row, O ig The final value of the element.

[0323] Thus, by correcting the output basic block of the k-th row through the above steps 1-3, the deviation introduced when the transpose of the key matrix was processed by the offset matrix can be eliminated, thereby ensuring the correctness of the attention calculation result.

[0324] Step 8035: Generate the output matrix based on the corrected multiple output basic blocks.

[0325] Since the k-th row of the output basic block is obtained by performing matrix multiplication between the k-th row of the weight basic block and the k-th row of the value basic block, after correcting the multiple output basic blocks, the corrected multiple output basic blocks are concatenated with reference to the arrangement and distribution of the multiple value basic blocks to obtain the output matrix.

[0326] In other words, the output matrix consists of N rows, and each row contains a modified output basic block.

[0327] Step 804: Determine the current output result of the network model based on the output matrix.

[0328] The output matrix contains the feature information of the current input information. Based on these extracted features, the network model can determine the current result.

[0329] It should be noted that the method for determining the current output result will vary depending on the current input information and the task performed by the network model. Specifically, the process of determining the current output result based on the features (i.e., the output matrix) of the current input information extracted by the attention mechanism module can be found in the inference flow of network models in related technologies, and will not be elaborated upon here in this application's embodiments.

[0330] In summary, for network models including attention mechanisms, the technical solution of this application processes the key matrix before performing matrix operations to obtain an intermediate matrix. Then, using the query matrix, intermediate matrix, and value matrix, the attention mechanism module determines the output matrix. Since the average value of each column element in the intermediate matrix is ​​less than the average value of the corresponding column elements in the transpose of the key matrix (i.e., the values ​​of elements in the intermediate matrix are attenuated relative to the transpose of the key matrix), numerical overflow and computational instability can be prevented at the computational source.

[0331] To facilitate understanding of the implementation process of the above scheme, the following will take the application scenario shown in Figure 6 above as an example, using the NPU as the underlying hardware, and supplement the explanation of the technical solution provided in the embodiments of this application based on the introduction of the FA (flash attention) operator and KV Cache technology.

[0332] Referring to Figure 14, in the method for determining the output result of the model provided in this application embodiment, the attention mechanism module in the network model can convert the current input information of the network model into corresponding input matrices Q, K, and V, and then use the processing method of the FA operator to divide Q, K, and V into blocks to obtain multiple basic blocks corresponding to the Q, K, and V matrices respectively.

[0333] It should be noted that the parameters regarding the size of the Q, K, and V matrices, as well as the size of the multiple basic blocks obtained from the partitioning, can be referred to the previous explanation and will not be repeated here.

[0334] In one possible implementation, an NPU is used to execute the technical solution provided in the embodiments of this application. The process of determining the output matrix based on the block results of the Q, K, and V matrices through the attention mechanism module may include the following steps (1)-(7).

[0335] (1) Load the parameters of the network model and allocate memory in the NPU.

[0336] This step (1) begins when the AI ​​framework calls the FA operator. In this step (1), the AI ​​framework (e.g., PyTorch) automatically splits the network and sets up parallelism. Then, it first reads the basic shape (i.e., the dimensions of input and output) and tiling information of the operator from the operator configuration file (e.g., .Json) and loads it into the HBM memory of the NPU.

[0337] The above step (1) is the same as the standard large model inference process. For relevant details, please refer to the large model inference process. It will not be repeated here.

[0338] (2) Construct the offset matrix based on the parameters of the network model.

[0339] The data types of the input matrix and information such as Tiling blocks are given in the configuration file at compile time. Therefore, the offset parameter α is a compile-time constant and resides in HBM memory.

[0340] Based on this offset parameter α, and K T Given the number of columns s² of the fundamental blocks of the matrix (i.e., the transpose of matrix K), construct an offset matrix. The elements of the main diagonal of the offset matrix are... The elements not on the main diagonal are

[0341] (3) Based on the offset matrix, for K T The matrix (i.e., the transpose of the K matrix) is preprocessed to obtain the intermediate matrix.

[0342] First, the NPU loads K from HBM(GM). T The matrix and offset matrix M are stored in the L1 cache. Then, K... T The matrix basic blocks are then loaded into the L0A cache, and the offset matrix M is loaded into the L0B cache, where the L0A and L0B caches are different spaces within the L0 cache. Further, batch matrix multiplication operations are performed through the Cube computation unit, i.e., K is calculated. T ×M is used to determine the intermediate matrix. Finally, the calculation result (intermediate matrix) is moved back to HBM(GM) through the Fixp data path to complete the preprocessing of the KV Cache.

[0343] See Figure 14, with K T Taking the basic block K1 in the matrix as an example, perform matrix multiplication on the basic block K1 and the offset matrix M to obtain the processed basic block K′1.

[0344] (4) Based on multiple query basic blocks and multiple intermediate basic blocks, multiple fractional basic blocks are obtained through matrix multiplication.

[0345] Step (4) is initiated after KV Cache preprocessing. The calculation process is consistent with the FA operator, and is accelerated by the Cube computing unit in the NPU. The overall implementation process is as follows: First, the NPU loads multiple query basic blocks of a single row of the Q matrix and multiple intermediate basic blocks of the preprocessed intermediate matrix from HBM(GM) into the L1 cache. Then, the multiple query basic blocks of the single row of the Q matrix are loaded into the L0A cache, and the multiple intermediate basic blocks of the intermediate matrix are loaded into the L0B cache. Further, matrix multiplication is performed through the Cube computing unit to obtain the corresponding fractional basic blocks. Finally, the calculated fractional basic blocks are moved back to HBM(GM) via the Fixp data path.

[0346] And so on, continue to perform the same operation on multiple query blocks in the next row of the Q matrix until the score blocks corresponding to all rows of the Q matrix are obtained.

[0347] Referring to Figure 14, taking the query basic block Q1 as an example, it is multiplied by matrix elements with the processed intermediate basic block K′1 to obtain the corresponding fractional basic block S1.

[0348] (5) Perform softmax calculation on multiple fractional basic blocks online to obtain multiple weighted basic blocks.

[0349] Considering the multi-pipeline architecture of the NPU, to improve the overall performance of the FA operator and in line with mainstream FA operators, softmax calculation can be performed online after the calculation of some fractional basic blocks is completed. The implementation process is as follows: First, the NPU loads the fractional basic blocks of the same row from HBM(GM) into UB. Then, the Vec computing unit calculates the maximum value of each element in the fractional basic block. This maximum value is used for scaling the exponential function in the subsequent softmax calculation to avoid numerical overflow. Further, the Vec computing unit calculates the weight basic blocks corresponding to the fractional basic blocks of that row. In addition, during the above calculation process, the Vec computing unit also needs to calculate the row sum of each element in multiple weight basic blocks. Finally, the calculation results—the maximum value of each element in the fractional basic block, the row sum of each element in the weight basic block, and the weight basic block itself—are loaded back from UB into HBM(GM).

[0350] This process continues, performing the same operation on multiple fractional blocks in the next row until the softmax calculation has been performed on all fractional blocks in all rows.

[0351] Referring to Figure 14, taking the fractional basic block S1 as an example, performing the above operations yields the corresponding weighted basic block P1 and the correction parameters. These correction parameters include the maximum value of each row element in each of the fractional basic blocks and the row sum of each row element in each of the weighted basic blocks.

[0352] (6) For the same row, use multiple fractional basic blocks of the row to calculate the sum of multiple global rows of the weighted basic blocks of the row.

[0353] Assuming that the first j-1 fractional basic blocks in the same row have completed the softmax calculation, and the maximum value of each row element of the first j-1 fractional basic blocks, the average value of each row element of the first j-1 fractional basic blocks, and the row sum of each row element of the first j-1 weighted basic blocks are obtained, these parameters are stored in the HBM(GM) and UB memory spaces.

[0354] Furthermore, the j-th fractional basic block is loaded from HBM(GM), and the multiple global row sums of the row weight basic block are determined by the above formulas (10)-(18).

[0355] (7) Perform offset correction on the output basic blocks.

[0356] Similarly, when the NPU corrects the output basic block of each row, it also needs to load the corresponding data from HBM(GM) into the Cube computing unit first, and after the correction is completed, move the result back to HBM(GM).

[0357] The process of offset correction of the output basic block can be referred to the above formulas (19)-(20), and will not be repeated here.

[0358] It should be noted that the above example content and the method embodiment shown in Figure 8 above belong to the same technical concept. For details on the corresponding implementation methods, calculation formulas, and beneficial effects, please refer to the above method embodiment, which will not be repeated here.

[0359] Figure 15 is a schematic diagram of a device for determining model output results according to an embodiment of this application. This device for determining model output results can be implemented by software, hardware, or a combination of both as part or all of the computer device shown in Figure 7. Referring to Figure 15, the network model includes an attention mechanism module, and the device 1500 for determining model output results includes: a first determining module 1501, a data processing module 1502, a second determining module 1503, and an output module 1504.

[0360] The first determining module 1501 is used to determine the query matrix, key matrix, and value matrix based on the current input information of the network model through the attention mechanism module; for detailed implementation process, please refer to the content of step 801 in the embodiment shown in Figure 8 above, which will not be repeated here.

[0361] The data processing module 1502 is used to process the transpose of the key matrix to obtain an intermediate matrix. The intermediate matrix and the transpose matrix have the same number of rows and columns. The average value of each column element in the intermediate matrix is ​​less than the average value of the corresponding column element in the transpose matrix. For detailed implementation process, please refer to step 802 in the embodiment shown in Figure 8 above. It will not be repeated here.

[0362] The second determining module 1503 is used to determine the output matrix based on the query matrix, intermediate matrix, and value matrix through the attention mechanism module; for detailed implementation process, please refer to the content of step 803 in the embodiment shown in Figure 8 above, which will not be repeated here.

[0363] The output module 1504 is used to determine the current output result of the network model based on the output matrix; for detailed implementation process, please refer to step 804 in the embodiment shown in Figure 8 above, which will not be repeated here.

[0364] In one possible implementation, the data processing module 1502 includes:

[0365] The parameter acquisition submodule is used to obtain offset parameters;

[0366] The matrix construction submodule is used to construct an offset matrix based on the offset parameter and the number of columns of the transpose matrix. All elements in the offset matrix are less than 1, the offset matrix has the same number of columns as the transpose matrix, and the number of rows and columns of the offset matrix are equal.

[0367] The matrix generation submodule is used to perform matrix multiplication on the transpose matrix and the offset matrix to obtain the intermediate matrix.

[0368] In one possible implementation, the transpose matrix comprises multiple transpose basic blocks, each containing elements with multiple rows and columns, and all elements in the multiple transpose basic blocks have the same number of rows and columns; the data processing module 1502 includes:

[0369] The parameter acquisition submodule is used to obtain offset parameters;

[0370] The matrix construction submodule is used to construct an offset matrix based on the offset parameter and the number of columns of the elements included in the transpose basic block. All elements in the offset matrix are less than 1, and the number of columns of the offset matrix is ​​the same as that of the elements included in the transpose basic block. The number of rows and columns of the offset matrix are also equal.

[0371] The matrix processing submodule is used to perform matrix multiplication operations between multiple transpose basic blocks and the offset matrix to obtain multiple intermediate basic blocks;

[0372] The matrix generation submodule is also used to generate intermediate matrices based on multiple intermediate basic blocks.

[0373] In one possible implementation, the elements of the main diagonal of the offset matrix are The elements not on the main diagonal are Where α is the offset parameter and s2 is the number of rows or columns of the offset matrix.

[0374] In one possible implementation, the query matrix includes multiple query basic blocks, the intermediate matrix includes multiple intermediate basic blocks, and the value matrix includes multiple value basic blocks; the second determining module 1503 includes:

[0375] The fraction calculation submodule is used to perform matrix multiplication on multiple query basic blocks and multiple intermediate basic blocks to obtain multiple fraction basic blocks;

[0376] The weight calculation submodule is used to determine multiple weight basic blocks based on multiple score basic blocks;

[0377] The output calculation submodule is used to determine multiple output basic blocks based on multiple weight basic blocks and multiple value basic blocks;

[0378] The correction submodule is used to perform offset correction on multiple output basic blocks to obtain corrected output basic blocks.

[0379] The output submodule is used to generate an output matrix based on multiple corrected output basic blocks.

[0380] In one possible implementation, multiple weight basic blocks are distributed in M ​​rows and N columns, where M and N are integers greater than or equal to 1. The weight basic blocks and output basic blocks have the same number of rows, and the weight basic blocks and output basic blocks also have the same number of rows. The correction submodule includes:

[0381] The row sum determination unit is used to determine multiple global row sums of the weight basic block in the k-th row of multiple weight basic blocks. The global row sum is the sum of a row of elements in the weight basic block in the k-th row, where k is greater than or equal to 1 and less than or equal to M.

[0382] The element correction unit is used to correct the i-th row element of the k-th row of the output basic block in multiple output basic blocks, so as to obtain the corrected i-th row element of the k-th row of the output basic block, where i is greater than or equal to 1 and less than or equal to the number of rows of elements included in the output basic block.

[0383] The global processing unit is used to divide each element of the corrected i-th row of the output basic block of the k-th row by the corresponding global row sum in multiple global row sums to obtain the corrected output basic block of the k-th row.

[0384] In one possible implementation, multiple value basic blocks are distributed in N rows, the number of columns of elements in the multiple value basic blocks is the same, the number of columns of elements in the multiple output basic blocks is the same, and the number of columns of elements in the value basic blocks and the output basic blocks is also the same; the element correction unit includes:

[0385] The row sum parameter determination sub-unit is used to set j=1 to determine the correction parameter of the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th row, and the correction parameter of the row sum of the i-th row elements of the j-th weight basic block. The row sum is the sum of the i-th row elements.

[0386] The element-value retrieval sub-unit is used to retrieve the values ​​of the O elements in the output basic block of the k-th row. ig The first and second values ​​of the element, O igAn element refers to the element located in the i-th row and g-th column of the k-th output basic block. The first value is determined based on the i-th row element of the first j-1 weight basic blocks and the g-th column element of the k-th value basic block. The second value is determined based on the i-th row element of the j-th weight basic block and the g-th column element of the k-th value basic block. g is greater than or equal to 1 and less than or equal to the number of columns of elements included in the value basic block.

[0387] The element correction subunit is used to determine the O values ​​in the output block of the k-th row based on the correction parameters of the first value, the second value, the row sum of the elements in the i-th row of the first j-1 weighted basic blocks, and the correction parameters of the row sum of the elements in the i-th row of the j-th weighted basic block. ig The third value of an element;

[0388] The first defined subunit is used to determine the O values ​​in the output basic block of the k-th row if j = N. ig The third value of the element is determined as the value of the element in the i-th row of the output basic block in the k-th row after correction;

[0389] The process of determining sub-units is also used to determine the correction parameters for the row sum of the first j-1 weight basic blocks in the k-th row, and the correction parameters for the row sum of the first i-th row of the weight basic block if j < N.

[0390] In one possible implementation, the row and determination unit includes:

[0391] The row sum parameter determination sub-unit is used to set j=1 to determine the correction parameter of the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th row, and the correction parameter of the row sum of the i-th row elements of the j-th weight basic block. The row sum is the sum of the i-th row elements.

[0392] The row sum acquisition sub-unit is used to acquire the row sum of the i-th row elements of the first j-1 weight basic blocks, and the row sum of the i-th row elements of the j-th weight basic block;

[0393] The row sum correction subunit is used to determine the corrected row sum of the i-th row elements of the first j-th weight basic blocks in the k-th weight basic block based on the correction parameters of the row sum of the i-th row elements of the first j-1 weight basic blocks, the correction parameters of the row sum of the i-th row elements of the first j-1 weight basic blocks, the row sum of the i-th row elements of the j-th weight basic block, and the row sum of the i-th row elements of the j-th weight basic block.

[0394] The second determining subunit is used to determine the global row sum of the i-th row elements of the first j weighted basic blocks after the correction of j = N.

[0395] The second determining subunit is also used to determine the correction parameters for the row sum of the i-th row elements of the first j-1 weight basic blocks in the k-th row, and the correction parameters for the row sum of the i-th row elements of the j-th weight basic block, if j < N.

[0396] In one possible implementation, multiple weight basic blocks correspond one-to-one with multiple fraction basic blocks, the fraction basic blocks contain the same number of rows of elements, and the fraction basic blocks and weight basic blocks contain the same number of rows of elements; the row and parameter determination subunit is also used for:

[0397] Get the correction parameter for the maximum value of the i-th row element of the first j-1 fractional basic blocks in the k-th row, and the correction parameter for the maximum value of the i-th row element of the j-th fractional basic block.

[0398] Get the maximum value of the i-th row of the first j-1 fractional basic blocks, the maximum value of the i-th row of the j-th fractional basic block, and the maximum value of the i-th row of the first j fractional basic blocks;

[0399] Based on the maximum value of the i-th row element of the first j-1 fractional basic blocks, the correction parameter of the maximum value of the i-th row element of the first j-1 fractional basic blocks, and the correction parameter of the maximum value of the i-th row element of the first j-1 fractional basic blocks, determine the correction parameter of the row sum of the i-th row element of the first j-1 weighted basic blocks.

[0400] Based on the maximum value of the i-th row element of the j-th fractional basic block, the maximum value of the i-th row element of the first j fractional basic blocks, and the correction parameter of the maximum value of the i-th row element of the j-th fractional basic block, determine the correction parameter of the row sum of the i-th row element of the j-th weighted basic block.

[0401] In one possible implementation, the row and parameter determination subunit is also used for:

[0402] Get the average of the i-th row of the first j-1 fractional basic blocks, the average of the i-th row of the j-th fractional basic block, and the average of the i-th row of the first j fractional basic blocks;

[0403] Get the offset parameter;

[0404] Based on the average value of the i-th row element of the first j-1 fractional basic blocks, the average value of the i-th row element of the j-th fractional basic block, the average value of the i-th row element of the first j-1 fractional basic blocks, and the offset parameter, determine the correction parameter for the maximum value of the i-th row element of the first j-1 fractional basic blocks and the correction parameter for the maximum value of the i-th row element of the j-th fractional basic block.

[0405] In one possible implementation, the row and parameter determination subunit is also used for:

[0406] The average value of the i-th row of the first j-1 fractional basic blocks is determined based on the average value of the i-th row of the first j-1 fractional basic blocks and the average value of the i-th row of the first j-1 fractional basic blocks.

[0407] In one possible implementation, the row and parameter determination subunit is also used for:

[0408] Based on the correction parameter of the maximum value of the i-th row element of the first j-1 fractional basic blocks, the maximum value of the i-th row element of the first j-1 fractional basic blocks is corrected to obtain the corrected maximum value of the i-th row element of the first j-1 fractional basic blocks.

[0409] Based on the correction parameter of the maximum value of the i-th row element of the j-th fractional basic block, the maximum value of the i-th row element of the j-th fractional basic block is corrected to obtain the corrected maximum value of the i-th row element of the j-th fractional basic block.

[0410] Determine the maximum value of the i-th row element of the first j-th fractional basic blocks from the maximum value of the i-th row element after correction of the first j-1 fractional basic blocks and the maximum value of the i-th row element of the j-th fractional basic block after correction.

[0411] In one possible implementation, the parameter acquisition unit is specifically used for:

[0412] Retrieves the data type of any matrix among the query matrix, key matrix, and value matrix. The data type indicates the floating-point precision of the data.

[0413] Based on the data type, the offset parameter is determined from the correspondence between the data type and the offset parameter.

[0414] In this embodiment, for a network model including an attention mechanism, the technical solution processes the key matrix before performing matrix operations to obtain an intermediate matrix. Then, using the query matrix, intermediate matrix, and value matrix, the attention mechanism module determines the output matrix. Since the average value of each column element in the intermediate matrix is ​​less than the average value of the corresponding column elements in the transpose of the key matrix (i.e., the values ​​of elements in the intermediate matrix are attenuated relative to the transpose of the key matrix), numerical overflow and computational instability can be prevented at the computational source.

[0415] It should be noted that the apparatus for determining the model output result provided in the above embodiments, when determining the output matrix through the attention mechanism module based on the current input information of the network model, and determining the current output result of the network model in combination with the output matrix, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for determining the model output result provided in the above embodiments and the method embodiments for determining the model output result belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0416] This application also provides a computer-readable storage medium storing instructions that, when executed on a processor, cause the processor to perform the method for determining the output result of the model as described in the above embodiments.

[0417] This application also provides a computer program product containing instructions that, when executed on a processor, cause the processor to perform the method for determining the output result of a model as described in the above embodiments. Alternatively, a computer program is provided that, when executed on a processor, causes the processor to perform the method for determining the output result of a model as described in the above embodiments.

[0418] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0419] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0420] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0421] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of determining a model output result, characterized by, The network model comprises an attention mechanism module; the method comprises: determining, by the attention mechanism module, a query matrix, a key matrix and a value matrix based on current input information of the network model; processing a transposed matrix of the key matrix to obtain an intermediate matrix, the intermediate matrix having the same number of rows and columns as the transposed matrix, and an average value of elements in each column of the intermediate matrix being less than an average value of corresponding elements in the transposed matrix; determining, by the attention mechanism module, an output matrix based on the query matrix, the intermediate matrix and the value matrix; determining a current output result of the network model based on the output matrix.

2. The method of claim 1, wherein, The processing of the transposed matrix of the key matrix to obtain the intermediate matrix comprises: obtaining an offset parameter; constructing an offset matrix based on the offset parameter and a column number of the transposed matrix, each element in the offset matrix being less than 1, the offset matrix having the same column number as the transposed matrix, and the offset matrix having the same number of rows and columns; performing matrix multiplication between the transposed matrix and the offset matrix to obtain the intermediate matrix.

3. The method of claim 1, wherein, The transposed matrix comprises a plurality of transposed basic blocks, each of the plurality of transposed basic blocks comprising a plurality of rows and a plurality of columns of elements, and each of the plurality of transposed basic blocks comprising the same number of rows and the same number of columns of elements; The processing of the transposed matrix of the key matrix to obtain the intermediate matrix comprises: obtain an offset parameter; construct an offset matrix based on the offset parameter and a column number of elements comprised by the transposed basic block, each element in the offset matrix being less than 1, the offset matrix having the same number of columns as the column number of elements comprised by the transposed basic block, and the offset matrix having the same number of rows and columns; perform matrix multiplication between each of the plurality of transposed basic blocks and the offset matrix to obtain a plurality of intermediate basic blocks; generate the intermediate matrix based on the plurality of intermediate basic blocks.

4. The method of claim 2 or 3, wherein, The elements of the main diagonal of the offset matrix are The elements of the non-main diagonal are where a is the offset parameter and s2 is the number of rows or columns of the offset matrix.

5. The method of claim 1 or 3, wherein, The query matrix comprises a plurality of query basic blocks, the intermediate matrix comprises a plurality of intermediate basic blocks, and the value matrix comprises a plurality of value basic blocks; The determination of the output matrix based on the query matrix, the intermediate matrix and the value matrix by the attention mechanism module comprises: performing matrix multiplication between the plurality of query basic blocks and the plurality of intermediate basic blocks to obtain a plurality of score basic blocks; determining a plurality of weight basic blocks based on the plurality of score basic blocks; determining a plurality of output basic blocks based on the plurality of weight basic blocks and the plurality of value basic blocks; performing offset correction on the plurality of output basic blocks to obtain a plurality of corrected output basic blocks; generating the output matrix based on the plurality of corrected output basic blocks.

6. The method of claim 5, wherein, The plurality of weight basic blocks are distributed in M rows and N columns, M and N are both integers greater than or equal to 1, the plurality of weight basic blocks comprise the same number of rows of elements, the plurality of output basic blocks comprise the same number of rows of elements, and the weight basic blocks and the output basic blocks comprise the same number of rows of elements; The offset correction on the plurality of output basic blocks to obtain the plurality of corrected output basic blocks comprises: determining a plurality of global row sums of the weight basic blocks in the kth row of the plurality of weight basic blocks, the global row sum being a sum of one row of elements of the weight basic blocks in the kth row, k being greater than or equal to 1 and less than or equal to the M; correcting the ith row element of the output basic blocks in the kth row of the plurality of output basic blocks to obtain a corrected ith row element of the output basic blocks in the kth row, i being greater than or equal to 1 and less than or equal to the number of rows of elements included in the output basic blocks; dividing the corrected ith row element of the output basic blocks in the kth row by the corresponding global row sum of the plurality of global row sums to obtain the corrected output basic blocks in the kth row.

7. The method of claim 6, wherein, The plurality of value basic blocks are arranged in N rows, the plurality of value basic blocks include the same number of columns of elements, the plurality of output basic blocks include the same number of columns of elements, and the value basic blocks and the output basic blocks include the same number of columns of elements. The correcting the ith row element of the output basic blocks in the kth row of the plurality of output basic blocks to obtain a corrected ith row element of the output basic blocks in the kth row includes: setting j = 1, determining a correction parameter of a row sum of the ith row element of the first j-1 weight basic blocks in the kth row of the plurality of weight basic blocks and a correction parameter of a row sum of the ith row element of the jth weight basic block, the row sum being a sum of the ith row element; Obtaining the first value and the second value of an element in the kth row of the output basic block ig The first value and the second value of an element are determined based on the ith row element of the first j-1 weight basic blocks and the gth column element of the value basic block in the kth row ig The first value and the second value of an element are determined based on the ith row element of the first j-1 weight basic blocks and the gth column element of the value basic block in the kth row The g is greater than or equal to 1 and less than or equal to the column number of the elements included in the value basic block determining a third value of the element based on the first value, the second value, a modified parameter of a row sum of the i-th row elements of the first j-1 weight base blocks, and a modified parameter of a row sum of the i-th row elements of the j-th weight base block; and ig the third value of the element. if j = N, then the value of the element in the output base block of the kth row O ig a third value of the element determined as the value of the modified ith row element of the output base block of the kth row; if j < N, setting j = j + 1, and returning to the step of determining the correction parameter of the row sum of the ith row element of the first j-1 weight basic blocks in the kth row of the plurality of weight basic blocks and the correction parameter of the row sum of the ith row element of the jth weight basic block.

8. The method of claim 6, wherein, The determining a plurality of global row sums of the weight basic blocks in the kth row of the plurality of weight basic blocks includes: setting j = 1, determining a correction parameter of a row sum of the ith row element of the first j-1 weight basic blocks in the kth row of the plurality of weight basic blocks and a correction parameter of a row sum of the ith row element of the jth weight basic block, the row sum being a sum of the ith row element; obtaining the row sum of the ith row element of the first j-1 weight basic blocks and the row sum of the ith row element of the jth weight basic block; determining a corrected row sum of the ith row element of the first j weight basic blocks in the kth row of the plurality of weight basic blocks based on the correction parameter of the row sum of the ith row element of the first j-1 weight weight basic blocks, the row sum of the ith row element of the first j-1 weight basic blocks, the correction parameter of the row sum of the ith row element of the jth weight basic block, and the row sum of the ith row element of the jth weight basic block; if j = N, determining the corrected row sum of the ith row element of the first j weight basic blocks as a global row sum of the ith row element of the first j weight basic blocks; if j < N, setting j = j + 1, and returning to the step for determining the correction parameter of the row sum of the ith row element of the first j-1 basic blocks in the kth row of the plurality of weight basic blocks and the correction parameter of a row sum of the ith row element of the jth weight basic block.

9. The method of claim 7 or 8, wherein, The plurality of weight basic blocks correspond to the plurality of fractional basic blocks one by one, the plurality of fractional basic blocks include the same number of rows of elements, and the fractional basic blocks also include the same number of rows of elements as the weight basic blocks; The determining the correction parameters of the row sum of the i-th row elements in the first j-1 weight basic blocks in the k-th row of weight basic blocks and the correction parameters of the row sum of the i-th row elements in the j-th weight basic block comprises: Obtaining the correction parameters of the maximum values of the i-th row elements in the first j-1 fractional basic blocks in the k-th row of fractional basic blocks and the correction parameters of the maximum values of the i-th row elements in the j-th fractional basic block; Obtaining the maximum values of the i-th row elements in the first j-1 fractional basic blocks, the maximum values of the i-th row elements in the j-th fractional basic block, and the maximum values of the i-th row elements in the first j fractional basic blocks; Based on the maximum values of the i-th row elements in the first j-1 fractional basic blocks, the maximum values of the i-th row elements in the first j fractional basic blocks, and the correction parameters of the maximum values of the i-th row elements in the first j-1 fractional basic blocks, the correction parameters of the row sum of the i-th row elements in the first j-1 weight basic blocks are determined; Based on the maximum values of the i-th row elements in the j-th fractional basic block, the maximum values of the i-th row elements in the first j fractional basic blocks, and the correction parameters of the maximum values of the i-th row elements in the j-th fractional basic block, the correction parameters of the row sum of the i-th row elements in the j-th weight basic block are determined.

10. The method of claim 9, wherein, The obtaining the correction parameters of the maximum values of the i-th row elements in the first j-1 fractional basic blocks in the k-th row of fractional basic blocks and the correction parameters of the maximum values of the i-th row elements in the j-th fractional basic block comprises: Obtaining the average values of the i-th row elements in the first j-1 fractional basic blocks, the average values of the i-th row elements in the j-th fractional basic block, and the average values of the i-th row elements in the first j fractional basic blocks; Obtaining the offset parameter; Based on the average values of the i-th row elements in the first j-1 fractional basic blocks, average values of the i-th row elements in the j-th fractional basic block, average values of the i-th row elements in the first j fractional basic blocks, and the offset parameter, the correction parameters of the maximum values of the i-th row elements in the first j-1 weight basic blocks and the correction parameters of the maximum values of the i-th row elements in the j-1 fractional basic block are determined.

11. The method of claim 10, wherein, The obtaining the average values of the i-th row elements in the first j fractional basic blocks comprises: Based on the average values of the i-th row elements in the first j-1 fractional basic block and the average values of the i-th row elements in the j-th fractional basic block, the average values of the i-th row elements in the first j fractional basic blocks are determined.

12. The method according to any one of claims 9 to 11, characterized in that, The obtaining the maximum values of the i-th row elements in the first j fractional basic blocks comprises: Based on the correction parameters of the maximum values of the i-th row elements in the first j-1 fraction basic blocks, the maximum values of the i-th row elements in the first j-1 fraction basic blocks are corrected to obtain the maximum values of the i-th row elements in the first j-1 fraction basic blocks after correction; correct the maximum value of the i-th row element of the j-th fractional basic block based on a correction parameter of the maximum value of the i-th row element of the j-th fractional basic block, to obtain a corrected maximum value of the i-th row element of the j-th fractional basic block; determine the maximum value of the i-th row element of the first j fractional basic blocks from the corrected maximum value of the i-th row element of the first j-1 fractional basic blocks and the corrected maximum value of the i-th row element of the j-th fractional basic block.

13. The method of claim 2, 3, or 10, wherein, The obtaining of the offset parameter comprises: obtaining a data type of any one of the query matrix, the key matrix and the value matrix, the data type indicating floating point precision of data; determining the offset parameter from a correspondence between data types and offset parameters based on the data type.

14. An apparatus for determining a model output result, the apparatus comprising: The network model comprises an attention mechanism module, and the apparatus comprises: a first determination module configured to determine, based on current input information of the network model, a query matrix, a key matrix and a value matrix through the attention mechanism module; a data processing module configured to process a transposed matrix of the key matrix to obtain an intermediate matrix, the intermediate matrix having the same number of rows and the same number of columns as the transposed matrix, and an average value of each column element in the intermediate matrix being smaller than an average value of a corresponding column element in the transposed matrix; a second determination module configured to determine, based on the query matrix, the intermediate matrix and the value matrix, an output matrix through the attention mechanism module; an output module configured to determine a current output result of the network model based on the output matrix.

15. The apparatus of claim 14, wherein, The data processing module comprises: a parameter acquisition sub-module configured to acquire an offset parameter; a matrix construction sub-module configured to construct an offset matrix based on the offset parameter and a number of columns of the transposed matrix, each element in the offset matrix being smaller than 1, the offset matrix having the same number of columns as the transposed matrix, and the offset matrix having the same number of rows and the same number of columns; a matrix generation sub-module configured to perform matrix multiplication between the transposed matrix and the offset matrix to obtain the intermediate matrix.

16. The apparatus of claim 14, wherein, The transposed matrix comprises a plurality of transposed basic blocks, each of the plurality of transposed basic blocks comprising a plurality of row elements and a plurality of column elements, and each of the plurality of transposed basic blocks comprising the same number of row elements and the same number of column elements; The data processing module comprises: a parameter acquisition sub-module configured to acquire an offset parameter; a parameter acquisition sub-module configured to acquire an offset parameter; a matrix construction sub-module configured to, based on the offset parameter and a number of columns of elements comprised by the transposed basic block, construct an offset matrix, each element in the offset matrix being smaller than 1, the offset matrix having the number of columns of elements comprised by the transposed basic block, and the offset matrix having the same number of rows and the same number of columns; a plurality of matrix processing sub-modules configured to perform matrix multiplication between each of the plurality of transposed basic blocks and the offset matrix to obtain a plurality of intermediate basic blocks; 17. The apparatus of claim 15 or 16, wherein, The elements of the main diagonal of the offset matrix are The elements of the non-main diagonal are where a is the offset parameter and s2 is the number of rows or columns of the offset matrix.

18. The apparatus of claim 14 or 16, wherein, The matrix generation sub-module is further configured to generate the intermediate matrix based on the plurality of intermediate basic blocks. The query matrix comprises a plurality of query basic blocks, the intermediate matrix comprises a plurality of intermediate basic blocks, and the value matrix comprises a plurality of value basic blocks. The second determination module comprises: The score calculation submodule is configured to perform matrix multiplication on the plurality of query basic blocks and the plurality of intermediate basic blocks to obtain a plurality of score basic blocks. The weight calculation submodule is configured to determine a plurality of weight basic blocks based on the plurality of score basic blocks. The output calculation submodule is configured to determine a plurality of output basic blocks based on the plurality of weight basic blocks and the plurality of value basic blocks. The correction submodule is configured to perform offset correction on the plurality of output basic blocks to obtain a plurality of corrected output basic blocks. The output submodule is configured to generate the output matrix based on the plurality of corrected output basic blocks.

19. The apparatus of claim 18, wherein, The plurality of weight basic blocks are distributed in M rows and N columns, M and N are integers greater than or equal to 1, the plurality of weight basic blocks include the same number of rows of elements, the plurality of output basic blocks include the same number of rows of elements, and the weight basic blocks and the output basic blocks include the same number of rows of elements. The correction submodule includes: The row sum determination unit is configured to determine a plurality of global row sums of the weight basic blocks in the kth row, the global row sum being a sum of elements in one row of the weight basic blocks in the kth row, k being greater than or equal to 1 and less than or equal to M. The element correction unit is configured to correct the ith row of elements of the output basic blocks in the kth row to obtain corrected ith row of elements of the output basic blocks in the kth row, i being greater than or equal to 1 and less than or equal to the number of rows of elements included in the output basic blocks. The global processing unit is configured to divide the corrected ith row of elements of the output basic blocks in the kth row by the corresponding global row sum in the plurality of global row sums to obtain the corrected output basic blocks in the kth row.

20. The apparatus of claim 19, wherein, The plurality of value basic blocks are distributed in N rows, the plurality of value basic blocks include the same number of columns of elements, the plurality of output basic blocks include the same number of columns of elements, and the value basic blocks and the output basic blocks include the same number of columns of elements. The element correction unit includes: The row sum parameter determination subunit is configured to set j = 1, determine a correction parameter of a row sum of the ith row of elements of the first j-1 weight basic blocks in the kth row of weight basic blocks, and a correction parameter of a row sum of the ith row of elements of the jth weight basic block, and the row sum being a sum of the ith row of elements. an element value obtaining subunit, configured to obtain an O ig a first value and a second value of an element, the O ig The element refers to an element located at an i-th row and a g-th column in the k-th row output basic block, the first value is determined based on an i-th row element of the first j-1 weight basic blocks and a g-th column element of the k-th row value basic block, the second value is determined based on an i-th row element of the j-th weight basic block and a g-th column element of the k-th row value basic block, and the g is greater than or equal to 1 and less than or equal to a column number of elements included in the value basic block. an element modification sub-unit configured to determine an output element in the k-th row of the output base block O ig a third value of the element; a first determining sub-unit, configured to determine, if j=N, O ig a third value of the element, determined as the value of the modified i-th row element of the output base block of the k-th row; The determination unit is further configured to, if j < N, set j = j + 1, and return to the step of determining the correction parameter of the row sum of the ith row of elements of the first j-1 weight basic blocks in the weight basic blocks in the kth row, and the correction parameter of the row sum of the ith row of elements of the jth weight basic block.

21. The apparatus of claim 19, wherein, The row sum determination unit includes: The row sum parameter determination subunit is configured to set j = 1, and determine a correction parameter of a row sum of the ith row of elements of the first j- 1 weight basic blocks in the kth row of weight basic blocks, and a correction parameter of the row sum of the ith row of elements of the jth weight basic block, and the row being a sum of the ith row of elements. The row sum acquisition subunit is configured to acquire the row sum of the ith row of elements of the first j-1 weight basic blocks, and the row sum of the ith row of elements of the jth weight basic block. a row sum and correction subunit configured to determine a corrected row sum of i-th row elements of the first j weight basis blocks based on a correction parameter of a row sum of i-th row elements of the first j-1 weight basis blocks, the row sum of i-th row elements of the first j-1 weight basis blocks, a correction parameter of a row sum of i-th row elements of the j-th weight basis block, and the row sum of i-th row elements of the j-th weight basis block; a second determination subunit configured to, if j=N, determine the corrected row sum of i-th row elements of the first j weight basis blocks as a global row sum of i-th row elements of the first j weight basis blocks; the second determination subunit is further configured to, if j 22. The apparatus of claim 20 or 21, wherein, the plurality of weight basis blocks correspond to the plurality of fractional basis blocks one by one, the plurality of fractional basis blocks include elements with the same row number, and the fractional basis blocks include elements with the same row number as the weight basis blocks; the row sum parameter determination subunit is further configured to: obtain a correction parameter of a maximum value of i-th row elements of the first j-1 fractional basis blocks in the k-th row of the plurality of fractional basis blocks and a correction parameter of a maximum value of i-th row elements of the j-th fractional basis block; obtain the maximum value of i-th row elements of the first j-1 fractional basis blocks, the maximum value of i-th row elements of the j-th fractional basis block, and the maximum value of i-th row elements of the first j fractional basis blocks; determine the correction parameter of the row sum of i-th row elements of the first j-1 weight basis blocks based on the maximum value of i-th row elements of the first j-1 fractional basis blocks, the correction parameter of the maximum value of i-th row elements of the first j fractional basis blocks, and the maximum value of i-th row elements of the first j-1 fractional basis blocks; determine the correction parameter of the row sum of i-th row elements of j-th weight basis block based on the maximum value of i-th row elements of the j-th fractional basis block, the correction parameter of the maximum value of i-th row elements of the first j fractional basis block, and the maximum value of i-th row elements of the j-th fractional basis block.

23. The apparatus of claim 22, wherein, the row sum parameter determination subunit is further configured to: obtain an average value of i-th row elements of the first j-1 fractional basis blocks, an average value of i-th row elements of the j-th fractional basis block, and an average value of i-th row elements of the first j fractional basis blocks; obtain an offset parameter; determine the correction parameter of the maximum value of i-th row elements of the first j-1 fractional basis blocks and the correction parameter of the maximum value of i-th row elements of the j-th fractional basis block based on the average value of i-th row elements of the first j-1 fractional basis blocks, the average value of i-th row elements of the j-th fractional basis block, the average value of i-th row elements of the first j fractional basis blocks, and the offset parameter.

24. The apparatus of claim 23, wherein, the row sum parameter determination subunit is further configured to: Determine an average value of the i-th row elements of the first j-1 fractional basic blocks based on an average value of the i-th row elements of the first j-1 fractional basic blocks and an average value of the i-th row elements of the j-th fractional basic block.

25. The apparatus of any one of claims 22-24, wherein, The row and parameter determination subunit is further configured to: Correct the maximum value of the i-th row elements of the first j-1 fractional basic blocks based on the correction parameter of the maximum value of the i-th row elements of the first j-1 fractional basic blocks to obtain a corrected maximum value of the i-th row elements of the first j-1 fractional basic blocks; Correct the maximum value of the i-th row elements of the j-th fractional basic block based on the correction parameter of the maximum value of the i-th row elements of the j-th fractional basic block to obtain a corrected maximum value of the i-th row elements of the j-th fractional basic block; Determine the maximum value of the i-th row elements of the first j fractional basic blocks from the corrected maximum value of the i-th row elements of the first j-1 fractional basic blocks and the corrected maximum value of the i-th row elements of the j-th fractional basic block.

26. The apparatus of claims 15, 16, or 23, wherein, The parameter acquisition unit is specifically configured to: Acquire a data type of any one of the query matrix, the key matrix and the value matrix, the data type indicating floating point precision of data; Determine the offset parameter from a correspondence between data types and offset parameters based on the data type.

27. A computer device, comprising: The computer device includes a processor and a memory; The memory is configured to store a computer program; The processor is configured to execute the computer program to implement the method in any one of claims 1-13.

28. A computer-readable storage medium, characterized in that, The storage medium has stored therein a computer program, and the computer program, when executed by a processor, implements the method in any one of claims 1-13.

29. A computer program product, characterised in that, The computer program product has stored therein computer instructions, and the computer instructions, when executed by a processor, implement the method in any one of claims 1-13.

Images