Model inference method and apparatus, and cluster, product and storage medium

By splitting the KV matrix on the computation nodes of the transformer model, the caching requirements are reduced, the problem of wasted caching resources on the computation nodes is solved, and more efficient caching utilization and accurate inference results are achieved.

WO2026232836A1PCT designated stage Publication Date: 2026-11-12HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2025/146350
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-08
Filing Date
2025-12-27
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

In the process of calculating the attention matrix of the transformer model, when the number of heads of the KV matrix and the number of heads of the Q matrix are in a one-to-many relationship, it leads to a waste of computing node cache resources.

Method used

By generating partial K and V matrices of the KV matrix on multiple computing nodes and using preset segmentation strategies such as batch dimension or sequence length dimension, the caching requirements of computing nodes are reduced, thus achieving accurate inference results.

Benefits of technology

It saves cache resources on computing nodes, improves cache utilization and the accuracy of inference results, and enhances the flexibility and accuracy of the method.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2025146350_12112026_PF_FP_ABST
    Figure CN2025146350_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The embodiments of the present application relate to the field of AI. Provided are a model inference method and apparatus, and a cluster, a product and a storage medium, which can save cache resources of computing nodes. The method comprises: receiving an inference request, and in response to the inference request, running a model to execute the following steps: generating a first KV matrix, wherein the first KV matrix comprises at least one head K matrix and at least one head V matrix; acquiring a head Q matrix set, wherein the head Q matrix set comprises a first head Q matrix and a second head Q matrix, the first head Q matrix is a head Q matrix that is generated by a first computing node, the second head Q matrix is a head Q matrix that is generated by a second computing node, and a KV matrix that is generated by the second computing node is the first KV matrix; and on the basis of a first K matrix, a first V matrix and the head Q matrix set, obtaining an inference result corresponding to the inference request, wherein the first K matrix is a partial matrix of the at least one head K matrix, and the first V matrix is a partial matrix of the at least one head V matrix.
Need to check novelty before this filing date? Find Prior Art

Description

A model inference method, apparatus, cluster, product, and storage medium.

[0001] This application claims priority to Chinese Patent Application No. 202510592451.7, filed with the State Intellectual Property Office of China on May 8, 2025, entitled “A Model Reasoning Method, Apparatus, Cluster, Product and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI), and more particularly to a model reasoning method, device, cluster, product, and storage medium. Background Technology

[0003] As is well known, in the process of calculating the attention matrix of a transformer model, it is necessary to generate query matrices (Q matrices), key matrices (K matrices), and value matrices (V matrices) with different heads; the number of heads in the K and V matrices is the same (kv head count). Then, the attention matrix is ​​calculated based on the Q, K, and V matrices.

[0004] However, when the number of KV (i.e., K matrix and V matrix) heads is one-to-many with the number of Q matrix heads, multiple computing nodes containing multiple Q matrix heads corresponding to a KV matrix A (i.e., K matrix A and V matrix A) need to cache the KV matrix A, resulting in high redundancy of the data cached by these multiple computing nodes. Therefore, this leads to a waste of the computing node's caching resources. Summary of the Invention

[0005] This application provides a model inference method, apparatus, cluster, product, and storage medium that can save cache resources on computing nodes.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a model inference method. The model performs inference based on a multi-head attention mechanism. The model is deployed on multiple computing nodes, and the method is applied to a first computing node among the multiple computing nodes. The method includes: receiving an inference request; and in response to the inference request, running the model to perform the following steps: generating a first KV matrix; wherein the first KV matrix includes at least one head K matrix from a plurality of head K matrices corresponding to the K matrix of the inference request, and at least one head V matrix from a plurality of head V matrices corresponding to the V matrix of the inference request; and obtaining a head Q matrix set; wherein the head Q matrix set includes: a first head Q matrix... The first head Q matrix is ​​generated by the first computation node among multiple head Q matrices, and the second head Q matrix is ​​generated by the second computation node among multiple head Q matrices. The multiple head Q matrices are multiple head matrices of the Q matrix corresponding to the inference request. Different computation nodes are used to generate different head Q matrices among the multiple head Q matrices. The KV matrix generated by the second computation node is the first KV matrix. Based on the first K matrix, the first V matrix and the head Q matrix set, the inference result corresponding to the inference request is obtained. The first K matrix is ​​a partial matrix in at least one head K matrix, and the first V matrix is ​​a partial matrix in at least one head V matrix.

[0008] The model inference method provided in this application is applied to a first computing node among multiple computing nodes. Responding to an inference request, the first computing node executes the following steps by running a model: generating a first KV matrix; wherein the first KV matrix includes at least one head K matrix of the K matrix corresponding to the inference request, and at least one head V matrix of the V matrix corresponding to the inference request. Obtaining a head Q matrix set; the head Q matrix set includes a first head Q matrix and a second head Q matrix; the first head Q matrix is ​​the head Q matrix generated by the first computing node among multiple head Q matrices, and the second head Q matrix is ​​the head Q matrix generated by the second computing node among multiple head Q matrices; the KV matrix generated by the second computing node is the first KV matrix. Then, based on a portion of the matrix in at least one head K matrix and a portion of the matrix in at least one head V matrix, and the head Q matrix set, the inference result corresponding to the inference request is obtained. It can be seen that in this application, the first computing node only needs to cache a portion of the K matrix and a portion of the V matrix in the first KV matrix, and does not need to cache the full data of the first KV matrix on the first computing node; therefore, it saves the caching resources of the computing node.

[0009] In one possible implementation, based on a preset segmentation strategy, the first KV matrix is ​​segmented into a target number of K matrices and a target number of V matrices; wherein the preset segmentation strategy includes: segmentation according to the batch dimension and / or segmentation according to the sequence length dimension; the target number is greater than or equal to 2; the target number of K matrices includes the first K matrix; the target number of V matrices includes the first V matrix; when the preset segmentation strategy includes segmentation according to the batch dimension, the first K matrix and the first V matrix belong to the same batch; when the preset segmentation strategy includes segmentation according to the sequence length dimension, the first K matrix and the first V matrix belong to the same position interval of the sequence.

[0010] This embodiment of the application divides the first KV matrix into a target number of K matrices and a target number of V matrices based on a preset segmentation strategy that segments according to the batch dimension and / or according to the sequence length dimension. Since the target number of K matrices includes the first K matrix and the target number of V matrices includes the first V matrix, and the inference result is subsequently obtained based on the first K matrix and the first V matrix, the first computing node does not need to cache the first KV matrix; it only needs to cache the first K matrix and the first V matrix. This reduces the amount of data in the first KV matrix cached on the first computing node, thus saving cache resources on the first computing node.

[0011] In one possible implementation, the target number is the number of computing nodes that generate the first KV matrix among multiple computing nodes.

[0012] The target data in this application can specifically be: the number of computing nodes among the multiple computing nodes where the first computing node resides that generate the same KV matrix; that is, the target data is the number of computing nodes that generate the same KV matrix; in other words, the target data is the sum of the number of second computing nodes and 1 (i.e., the first computing node). This allows different computing nodes to determine the inference result based on different K matrices and different V matrices from the target number of K matrices and target number of V matrices; it solves the problem of redundant storage of KV matrices on different computing nodes, thus improving the cache utilization rate in computing nodes.

[0013] In one possible implementation, the method further includes: obtaining the identifier of the first computing node; determining the first K matrix from a target number of K matrices based on the identifier of the first computing node; and determining the first V matrix from a target number of V matrices based on the identifier of the first computing node.

[0014] Based on the identifier of the first computing node, the first K matrix and the first V matrix are determined from the target number of K matrices and the target number of V matrices, respectively. This ensures that multiple computing nodes that generate the same KV matrix can obtain their own inference results based on different K matrices and V matrices, thus improving the accuracy of the inference results.

[0015] In one possible implementation, when the preset segmentation strategy includes segmentation according to the batch dimension, the head Q matrix set specifically includes: a first batch of head Q matrices and a second batch of head Q matrices; wherein, the first batch of head Q matrices is in the first head Q matrix and belongs to the same batch as the first K matrix and the first V matrix; the second batch of head Q matrices is in the second head Q matrix and belongs to the same batch as the first K matrix and the first V matrix.

[0016] In one possible implementation, obtaining the head Q matrix set includes: generating a first head Q matrix; and performing a batch-dimensional matrix swap on the first head Q matrix and the second head Q matrix generated by the second computing node based on a first set communication primitive to obtain the head Q matrix set.

[0017] This embodiment of the application determines the first K matrix and the first V matrix by partitioning the first KV matrix according to the batch dimension. Then, based on the first set communication primitive, a batch-dimensional matrix exchange is performed on the first head Q matrix generated by the first computing node and the first head Q matrix generated by the second computing node, so that the batch of head Q matrices included in the head Q matrix set in the first computing node is consistent with the batch of the first K matrix and the first V matrix; thereby ensuring the accuracy of the first attention matrix determined based on the first K matrix, the first V matrix and the head Q matrix set.

[0018] In one possible implementation, when the preset segmentation strategy includes segmentation according to the sequence length dimension, obtaining the head Q matrix set includes: generating a first head Q matrix; performing full synchronization on the first head Q matrix and the second head Q matrix generated by the second computing node based on the second set communication primitive to obtain the head Q matrix set; the synchronized first computing node and the second computing node both include the head Q matrix set.

[0019] This embodiment of the application determines the first K matrix and the first V matrix by segmenting the first KV matrix according to the sequence length dimension. Then, based on the second set communication primitive, full synchronization is performed on the first head Q matrix and the second head Q matrix generated by the second computing node. Therefore, both the synchronized first computing node and the second computing node include the first head Q matrix and the second head Q matrix. This ensures that the first head Q matrix and the second head Q matrix are each operated on with matrices from different sequence intervals of at least one head K matrix, thereby guaranteeing the accuracy of the operation results.

[0020] In one possible implementation, obtaining the inference result corresponding to the inference request based on the first K matrix, the first V matrix, and the head Q matrix set includes: determining a first attention matrix based on the first K matrix, the first V matrix, and the head Q matrix set; wherein, when the preset segmentation strategy includes segmentation according to the batch dimension, the first attention matrix includes: a first sub-Q attention matrix and a second sub-Q attention matrix; the first sub-Q attention matrix is ​​the attention matrix corresponding to the first batch of head Q matrices, and the second sub-Q attention matrix is ​​the attention matrix corresponding to the second batch of head Q matrices; the first batch of head Q matrices are matrices in the first head Q matrix that belong to the same batch as the first K matrix and the first V matrix; the second batch of head Q matrices... The matrix is ​​part of the second-head Q matrix and belongs to the same batch as the first K matrix and the first V matrix. Under the preset segmentation strategy, which includes segmentation according to sequence length, the first attention matrix includes: a first sub-KV attention matrix and a second sub-KV attention matrix. The first sub-KV attention matrix is ​​the attention matrix corresponding to the first-head Q matrix, the first K matrix, and the first V matrix; the second sub-KV attention matrix is ​​the attention matrix corresponding to the second-head Q matrix, the first K matrix, and the first V matrix. Based on the first attention matrix, the target attention matrix is ​​obtained; where the target attention matrix is ​​the attention matrix corresponding to the first-head Q matrix and the first KV matrix. Based on the target attention matrix, the inference result is obtained.

[0021] This embodiment of the application divides the first KV matrix into a target number of K matrices and a target number of V matrices based on a preset segmentation strategy that segments according to the batch dimension and / or the sequence length dimension. Then, based on the identifier of the first computing node, the first K matrix and the first V matrix are determined from the target number of K matrices and the target number of V matrices, respectively. Then, based on the first K matrix, the first V matrix, and the head Q matrix set, a first attention matrix is ​​determined; subsequently, the first computing node obtains the target attention matrix based on the first attention matrix, and obtains the inference result based on the target attention matrix. Since the first K matrix is ​​a partial matrix in at least one head K matrix, and the first V matrix is ​​a partial matrix in at least one head V matrix, it is not necessary to cache the full data of the first KV matrix on the first computing node; therefore, the caching resources of the computing node are saved.

[0022] In one possible implementation, when the preset segmentation strategy includes segmentation according to the batch dimension, the target attention matrix is ​​obtained based on the first attention matrix, including: performing a head-dimensional matrix exchange between the first attention matrix and the second attention matrix generated by the second computing node based on the first set communication primitive to obtain the target attention matrix; wherein, the second attention matrix includes: a third sub-Q attention matrix and a fourth sub-Q attention matrix; the third sub-Q attention matrix is ​​the attention matrix corresponding to the third batch of head Q matrices, and the third batch of head Q matrices are matrices of other batches in the first head Q matrix excluding the first batch of head Q matrices; the fourth sub-Q attention matrix is ​​the attention matrix corresponding to the fourth batch of head Q matrices, and the fourth batch of head Q matrices are matrices of other batches in the second head Q matrix excluding the second batch of head Q matrices; the target attention matrix includes: the first sub-Q attention matrix and the third sub-Q attention matrix.

[0023] This embodiment of the application, based on a first set of communication primitives, performs a batch-level matrix exchange on the first head Q matrix generated by the first computing node and the first head Q matrix generated by the second computing node. After determining the first attention matrix based on the first K matrix, the first V matrix, and the head Q matrix set, a head-level matrix exchange is performed on the first attention matrix and the second attention matrix generated by the second computing node, based on the first set of communication primitives, to obtain the target attention matrix. The second attention matrix includes a third sub-Q attention matrix and a fourth sub-Q attention matrix. The third sub-Q attention matrix corresponds to the attention matrix of the third batch of head Q matrices, which are matrices from other batches of the first head Q matrices besides the first batch. The fourth sub-Q attention matrix corresponds to the attention matrix of the fourth batch of head Q matrices, which are matrices from other batches of the second head Q matrices besides the second batch. Furthermore, the target attention matrix includes both the first sub-Q attention matrix and the third sub-Q attention matrix; therefore, the target attention matrix includes the attention matrix corresponding to the first head Q matrix. Subsequently, the first computing node obtains the inference result based on this target attention matrix, thereby improving the accuracy of the inference result.

[0024] In one possible implementation, when the preset segmentation strategy includes segmentation according to the sequence length dimension, the target attention matrix is ​​obtained based on the first attention matrix, including: performing a head-dimensional matrix exchange between the first attention matrix and the second attention matrix generated by the second computing node based on the third set communication primitive to obtain an intermediate attention matrix; wherein, the second attention matrix includes: a third sub-KV attention matrix and a fourth sub-KV attention matrix; the third sub-KV attention matrix is ​​the attention matrix corresponding to the first head Q matrix, the second K matrix, and the second V matrix, and the fourth sub-KV attention matrix is ​​the attention matrix corresponding to the second head Q matrix, the second K matrix, and the second V matrix; wherein, the second K matrix is ​​a matrix other than the first K matrix in at least one head K matrix, and the second V matrix is ​​a matrix other than the first V matrix in at least one head V matrix; the intermediate attention matrix includes: the first sub-KV attention matrix and the third sub-KV attention matrix; concatenating the attention matrices with the same batch in the first sub-KV attention matrix and the third sub-KV attention matrix yields the target attention matrix.

[0025] In this embodiment, the first attention matrix determined by the first computing node and the second attention matrix generated by the second computing node are swapped at the head dimension based on a third set communication primitive. This allows the first computing node to obtain attention matrices corresponding to different sequence length intervals in the first KV matrix and the first head Q matrix (i.e., the first sub-KV attention matrix and the third sub-KV attention matrix), and the second computing node to obtain attention matrices corresponding to different sequence length intervals in the first KV matrix and the second head Q matrix (i.e., the second sub-KV attention matrix and the fourth sub-KV attention matrix). Then, the first computing node concatenates the attention matrices with the same batch in the first sub-KV attention matrix and the third sub-KV attention matrix to obtain the target attention matrix. Subsequently, the first node obtains the inference result based on this target attention matrix, thereby improving the accuracy of the inference result.

[0026] In one possible implementation, before obtaining the head Q matrix set, the method further includes: outputting a first interface; the first interface includes a splitting option and a non-splitting option; obtaining the target option selected by the user from the first interface; obtaining the head Q matrix set includes: obtaining the head Q matrix set when the target option is a splitting option.

[0027] This application embodiment improves the flexibility of executing the model reasoning method provided by this application embodiment by obtaining the target option selected by the user from the first interface and determining whether to execute the method provided by this application embodiment.

[0028] In one possible implementation, the segmentation options include at least one of a first segmentation option, a second segmentation option, and an adaptive segmentation option; the first segmentation option indicates that the preset segmentation strategy is segmentation according to the batch dimension; the second segmentation option indicates that the preset segmentation strategy is segmentation according to the sequence length dimension; the adaptive segmentation option indicates that: when the batch size of the first KV matrix is ​​greater than a first threshold and the sequence length of the first KV matrix is ​​less than a second threshold, the preset segmentation strategy includes segmentation according to the batch dimension; when the batch size of the first KV matrix is ​​less than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes segmentation according to the sequence length dimension; when the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes both segmentation according to the batch dimension and segmentation according to the sequence length dimension.

[0029] The segmentation options in the first interface of this application embodiment include at least one of a first segmentation option, a second segmentation option, and an adaptive segmentation option. The first segmentation option indicates that the preset segmentation strategy is segmentation according to the batch dimension; the second segmentation option indicates that the preset segmentation strategy is segmentation according to the sequence length dimension. The adaptive segmentation option indicates that: when the batch size of the first KV matrix is ​​greater than a first threshold and the sequence length of the first KV matrix is ​​less than a second threshold, the preset segmentation strategy includes segmentation according to the batch dimension; when the batch size of the first KV matrix is ​​less than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes segmentation according to the batch dimension and segmentation according to the sequence length dimension; when the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes both segmentation according to the batch dimension and segmentation according to the sequence length dimension. Thus, the preset segmentation strategy is determined by the user's selection in the first interface, thereby improving the flexibility of the preset segmentation strategy.

[0030] In one possible implementation, the method further includes: deleting at least one matrix in the head K matrix other than the first K matrix; and deleting at least one matrix in the head V matrix other than the first V matrix.

[0031] In this embodiment of the application, after determining the first K matrix, the method further includes: the first computing node deleting at least one matrix other than the first K matrix from at least one head K matrix (i.e., the target number of K matrices); correspondingly, the first computing node deleting at least one matrix other than the first V matrix from at least one head V matrix (i.e., the target number of V matrices). This reduces the amount of data in the first KV matrix cached on the first computing node, thus saving cache resources on the first computing node.

[0032] Secondly, embodiments of this application provide a model inference apparatus. The model performs inference based on a multi-head attention mechanism. The model is deployed on multiple computing nodes, and the model inference apparatus is applied to a first computing node among the multiple computing nodes. The model inference apparatus includes: a transceiver module, a running module, a generation module, and an inference module. The transceiver module is used to receive inference requests. The running module is used to respond to the inference request and run the model to perform the following steps: the generation module is used to generate a first KV matrix; wherein, the first KV matrix includes: at least one head K matrix among multiple head K matrices of the K matrix corresponding to the inference request, and at least one head V matrix among multiple head V matrices of the V matrix corresponding to the inference request; the transceiver module is also used to obtain... Take a set of head Q matrices; wherein the set of head Q matrices includes: a first head Q matrix and a second head Q matrix; the first head Q matrix is ​​the head Q matrix generated by the first computing node among multiple head Q matrices, the second head Q matrix is ​​the head Q matrix generated by the second computing node among multiple head Q matrices, and the multiple head Q matrices are multiple head matrices of the Q matrix corresponding to the inference request; different computing nodes are used to generate different head Q matrices among the multiple head Q matrices, and the KV matrix generated by the second computing node is the first KV matrix; the inference module is used to obtain the inference result corresponding to the inference request based on the first K matrix, the first V matrix and the set of head Q matrices; the first K matrix is ​​a partial matrix in at least one head K matrix, and the first V matrix is ​​a partial matrix in at least one head V matrix.

[0033] In one implementation, the model inference device further includes: a segmentation module; the segmentation module is used to segment the first KV matrix into a target number of K matrices and a target number of V matrices based on a preset segmentation strategy; wherein the preset segmentation strategy includes: segmentation according to the batch dimension and / or segmentation according to the sequence length dimension; the target number is greater than or equal to 2; the target number of K matrices includes the first K matrix; the target number of V matrices includes the first V matrix; when the preset segmentation strategy includes segmentation according to the batch dimension, the first K matrix and the first V matrix belong to the same batch; when the preset segmentation strategy includes segmentation according to the sequence length dimension, the first K matrix and the first V matrix belong to the same position interval of the sequence.

[0034] In one implementation, the target number is the number of computing nodes that generate the first KV matrix among multiple computing nodes.

[0035] In one implementation, the transceiver module is used to obtain the identifier of the first computing node; the inference module is used to determine the first K matrix from the target number of K matrices based on the identifier of the first computing node; the inference module is also used to determine the first V matrix from the target number of V matrices based on the identifier of the first computing node.

[0036] In one implementation, when the preset segmentation strategy includes segmentation according to the batch dimension, the head Q matrix set specifically includes: a first batch of head Q matrices and a second batch of head Q matrices; wherein, the first batch of head Q matrices is in the first head Q matrix and belongs to the same batch as the first K matrix and the first V matrix; the second batch of head Q matrices is in the second head Q matrix and belongs to the same batch as the first K matrix and the first V matrix.

[0037] In one implementation, the generation module generates a first head Q matrix; the inference module performs a batch-dimensional matrix swap on the first head Q matrix and the second head Q matrix generated by the second computing node based on a first set communication primitive, to obtain a head Q matrix set.

[0038] In one implementation, the generation module generates a first head Q matrix; the inference module performs full synchronization of the first head Q matrix and the second head Q matrix generated by the second computing node based on the second set communication primitive, to obtain a head Q matrix set; the synchronized first computing node and the second computing node both include the head Q matrix set.

[0039] In one implementation, the inference module determines a first attention matrix based on a first K matrix, a first V matrix, and a set of head Q matrices. Wherein, if the preset segmentation strategy includes segmentation by batch dimension, the first attention matrix includes: a first sub-Q attention matrix and a second sub-Q attention matrix; the first sub-Q attention matrix is ​​the attention matrix corresponding to the first batch of head Q matrices, and the second sub-Q attention matrix is ​​the attention matrix corresponding to the second batch of head Q matrices; the first batch of head Q matrices are those within the first head Q matrix and belong to the same batch as the first K matrix and the first V matrix; the second batch of head Q matrices are those within the second head Q matrix and belong to the same batch as the first K matrix and the first V matrix. The first attention matrix includes: a first sub-KV attention matrix and a second sub-KV attention matrix, provided that the preset segmentation strategy includes segmentation according to the sequence length dimension. The first sub-KV attention matrix is ​​the attention matrix corresponding to the first head Q matrix, the first K matrix, and the first V matrix, and the second sub-KV attention matrix is ​​the attention matrix corresponding to the second head Q matrix, the first K matrix, and the first V matrix. The inference module is also used to obtain a target attention matrix based on the first attention matrix. The target attention matrix is ​​the attention matrix corresponding to the first head Q matrix and the first KV matrix. The inference module is also used to obtain an inference result based on the target attention matrix.

[0040] In one implementation, the inference module performs a head-dimensional matrix swap between the first attention matrix and the second attention matrix generated by the second computing node based on a first set of communication primitives to obtain a target attention matrix. The second attention matrix includes a third sub-Q attention matrix and a fourth sub-Q attention matrix. The third sub-Q attention matrix is ​​the attention matrix corresponding to the third batch of head-Q matrices, which are matrices from other batches of the first head-Q matrices besides the first batch. The fourth sub-Q attention matrix is ​​the attention matrix corresponding to the fourth batch of head-Q matrices, which are matrices from other batches of the second head-Q matrices besides the second batch. The target attention matrix includes the first sub-Q attention matrix and the third sub-Q attention matrix.

[0041] In one implementation, the inference module performs a head-dimensional matrix exchange between the first attention matrix and the second attention matrix generated by the second computing node based on a third set communication primitive to obtain an intermediate attention matrix. The second attention matrix includes a third sub-KV attention matrix and a fourth sub-KV attention matrix. The third sub-KV attention matrix is ​​the attention matrix corresponding to the first head Q matrix, the second K matrix, and the second V matrix; the fourth sub-KV attention matrix is ​​the attention matrix corresponding to the second head Q matrix, the second K matrix, and the second V matrix. The second K matrix is ​​any matrix other than the first K matrix in at least one head K matrix, and the second V matrix is ​​any matrix other than the first V matrix in at least one head V matrix. The intermediate attention matrix includes the first sub-KV attention matrix and the third sub-KV attention matrix. The inference module concatenates the attention matrices with the same batch in the first sub-KV attention matrix and the third sub-KV attention matrix to obtain the target attention matrix.

[0042] In one implementation, the transceiver module is used to output a first interface; the first interface includes a splitting option and a non-splitting option; the transceiver module is used to obtain the target option selected by the user from the first interface; the above-mentioned obtaining the head Q matrix set includes: the transceiver module is used to obtain the head Q matrix set when the target option is a splitting option.

[0043] In one implementation, the segmentation options include at least one of a first segmentation option, a second segmentation option, and an adaptive segmentation option; the first segmentation option indicates that the preset segmentation strategy is segmentation according to the batch dimension; the second segmentation option indicates that the preset segmentation strategy is segmentation according to the sequence length dimension; the adaptive segmentation option indicates that: when the batch size of the first KV matrix is ​​greater than a first threshold and the sequence length of the first KV matrix is ​​less than a second threshold, the preset segmentation strategy includes segmentation according to the batch dimension; when the batch size of the first KV matrix is ​​less than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes segmentation according to the sequence length dimension; when the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes both segmentation according to the batch dimension and segmentation according to the sequence length dimension.

[0044] In one implementation, the inference module is used to delete at least one matrix in the head K matrix other than the first K matrix; the inference module is used to delete at least one matrix in the head V matrix other than the first V matrix.

[0045] Thirdly, this application provides a computing device cluster including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method described in the first aspect and any of its possible implementations.

[0046] Fourthly, this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed on a computing device, cause the computing device to perform the method described in any one of the first aspects and its possible implementations.

[0047] Fifthly, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in any one of the first aspects and their possible implementations.

[0048] It should be understood that the beneficial effects of the technical solutions of the second to fifth aspects of this application and the corresponding possible implementations can be referred to the above-described technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0049] Figure 1 is a schematic diagram of a cloud service architecture provided in this application;

[0050] Figure 2 is a schematic diagram of a model management system provided in this application;

[0051] Figure 3 is one of the schematic diagrams of a model reasoning method provided in this application;

[0052] Figure 4 is a second schematic diagram of a model reasoning method provided in this application;

[0053] Figure 5 is a schematic diagram of a batch dimension segmentation provided in this application;

[0054] Figure 6 is a schematic diagram of a sequence length dimension segmentation provided in this application;

[0055] Figure 7 is a schematic diagram of batch dimension segmentation and sequence length dimension segmentation provided in this application;

[0056] Figure 8 is a schematic flowchart of one of the methods for determining the attention matrix provided in this application;

[0057] Figure 9 is a second schematic diagram of a method for determining an attention matrix provided in this application;

[0058] Figure 10 is a third schematic diagram of a method for determining an attention matrix provided in this application;

[0059] Figure 11 is a schematic diagram of a method for obtaining a preset segmentation strategy provided in this application;

[0060] Figure 12 is a schematic diagram of a first interface provided in this application;

[0061] Figure 13 is a schematic diagram of another first interface provided in this application;

[0062] Figure 14 is a schematic diagram of the structure of a model reasoning device provided in this application;

[0063] Figure 15 is a hardware schematic diagram of a computing device provided in this application;

[0064] Figure 16 is a schematic diagram of a computing device cluster provided in this application;

[0065] Figure 17 is a schematic diagram of a network connection provided in an embodiment of this application. Detailed Implementation

[0066] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0067] The terms "first head Q matrix" and "second head Q matrix" in the specification and claims of the embodiments of this application are used to distinguish different head Q matrices, rather than to describe a specific order of the head Q matrices. For example, the first K matrix and the second K matrix are used to distinguish different K matrices, rather than to describe a specific order of the K matrices.

[0068] In the embodiments of this application, words such as "exemplary" or "for example" are used to denote an example, illustration or description. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application shall not be construed as being more preferred or advantageous than other embodiments or design solutions. To be precise, the use of "exemplary" or "for example" is intended to present the related concepts in a specific manner.

[0069] In the description of the embodiments of this application, unless otherwise specified, "a plurality of" means two or more. For example, a plurality of computing nodes refers to two or more computing nodes.

[0070] First, some concepts involved in the model inference method, apparatus, cluster, product and storage medium provided by the embodiments of this application are explained, as follows specifically:

[0071] Multi-head attention mechanism: it is an extended form of attention mechanism widely adopted in transformer models, which obtains attention distributions of different subspaces of an input sequence by running multiple independent attention mechanisms in parallel, so as to more comprehensively capture various potential semantic associations in the sequence.

[0072] Semantic space: it is an abstract mathematical space used in natural language processing to represent text semantic information, which captures the semantic relevance of vocabulary, sentences or paragraphs in the form of vectors or matrices. For example, the character "good" has different semantics in the two semantic spaces of "Li Ming is a good person" and "Li Ming is eager to learn".

[0073] Head matrix: also referred to as a head, which means splitting a matrix into a plurality of sub-matrices according to semantic spaces; wherein each sub-matrix in the plurality of sub-matrices is a head matrix.

[0074] For example: the high-dimensional matrix corresponding to the input sequence "I am an aviator" is a matrix comprising 4 semantic spaces (also called: contexts); then the matrix is a (4,5) matrix (i.e., a matrix with 4 rows and 5 columns). Wherein each row in the (4,5) matrix corresponds to the weight of a character in the 4 semantic spaces (for example, the first row is the weight of the character "I" in the 4 semantic spaces). The (4,5) matrix is split into 2 (2,5) sub-matrices according to columns (i.e., semantic spaces); then each (2,5) sub-matrix is a head matrix of the (4,5) matrix.

[0075] Attention matrix: It is the core data structure in the attention mechanism, used to dynamically quantify the correlation strength between different elements in the input sequence. Weights are generated by calculating the similarity between the Q matrix and the K matrix, and then weighted and aggregated into a V matrix to form a dynamically focused context representation.

[0076] The encoder stage is used to extract the global contextual representation of the input sequence (e.g., the source language sentence "I am an aviator") and generate a high-dimensional semantic matrix (i.e., a high-dimensional language vector).

[0077] Decoding stage: Based on the target sequence output from the encoding stage (i.e., the high-dimensional semantic matrix corresponding to the input sequence), and combined with the context of the input sequence, an output sequence (i.e., the inference result) is generated.

[0078] With the rapid development of AI technology, there are more and more models that perform inference based on multi-head attention mechanisms (such as the transformer model).

[0079] A typical model inference process includes: obtaining an inference request; then, distributing the input sequence from the inference request (e.g., "I am an aviator") to multiple compute nodes where the model is deployed. Each compute node then generates a Q-matrix, a K-matrix, and a V-matrix based on the sequence; the Q-matrix generated by different compute nodes has different heads. Subsequently, each compute node generates its own attention matrix based on its generated Q-matrix, K-matrix, and V-matrix. Finally, the inference result is generated based on the multiple attention matrices generated by the multiple compute nodes.

[0080] However, when the number of headers of the KV matrices (i.e., K matrices and V matrices) generated by multiple computing nodes is one-to-many with the number of headers of the Q matrices generated by multiple computing nodes, at least two computing nodes have the same KV matrices. This causes these at least two computing nodes to repeatedly cache the same KV matrices, resulting in high redundancy of data cached by multiple computing nodes. Therefore, this leads to a waste of computing node caching resources.

[0081] Based on this, this application provides a model inference method. When the KV matrices generated by the first computing node and the second computing node are the same, the first computing node obtains the inference result based on a portion of the K and V matrices in the KV matrix, a first head Q matrix generated by the first computing node, and a second head Q matrix generated by the second computing node. In other words, in this application, the first computing node only needs to cache a portion of the K and V matrices in the KV matrix, and does not need to cache the full data of the KV matrix; therefore, the caching resources of the computing node are saved.

[0082] The model inference method provided in this application can be applied to ordinary servers as well as cloud servers. The specific embodiments of this application do not limit the application scenarios of the model inference method provided in this application.

[0083] The model inference method provided in this application is specifically applied in the decoding stage of model operation.

[0084] For example, the model inference method provided in this application embodiment is applied to the cloud service architecture shown in Figure 1; the cloud server architecture includes: clients and data centers.

[0085] The data center includes a cloud management platform and multiple servers (Server 1 and Server 2 as shown in Figure 1); the cloud management platform is connected to the multiple servers through the data center's internal network.

[0086] The client connects to the cloud management platform via the internet; the client is used to send instructions to multiple servers through the cloud management platform. For example, in this application, the client sends inference requests to multiple servers through the cloud management platform.

[0087] Each of the multiple servers comprises a hardware layer and a software layer. The hardware layer includes hardware components such as memory, processor, network interface card, and disk, which are connected via a bus to provide basic computing and storage capabilities for the upper-layer software.

[0088] The software layer includes the host operating system and multiple virtual machines (e.g., virtual machine 1 and virtual machine 2); these multiple virtual machines are logical computing environments created on the host operating system and can run applications independently.

[0089] The host operating system includes a virtual machine manager and a cloud management platform client. The virtual machine manager is responsible for managing the lifecycle operations of virtual machines running on the host operating system, such as creation, startup, and shutdown.

[0090] The cloud management platform client is used to communicate with an external cloud management platform, receive and execute management instructions issued by the platform. For example, the cloud management platform client receives inference requests sent by the client through the cloud management platform and controls the computing nodes to execute the model inference method provided in the embodiments of this application.

[0091] It is worth noting that Figure 1 is only a schematic diagram and should not be construed as a limitation of this application. The cloud service architecture may include more or fewer devices, and this application does not limit this.

[0092] The model inference method provided in this application is applied to a model that performs inference based on a multi-head attention mechanism. This model (e.g., an AI model) is deployed on the server shown in Figure 1, or a virtual machine within that server. The model management system shown in Figure 2 includes a cloud environment for deploying the model and a client for accessing the AI ​​model.

[0093] Each node in the cloud environment (e.g., any node from node 1 to node N) is a server as shown in Figure 1, or a virtual machine in a server; and each node independently deploys the model (i.e., the AI ​​model shown in Figure 2).

[0094] Clients accessing AI models can be either client / server (C / S) or browser / server (B / S) architecture applications. Users access AI models online through the client; specifically, users input inference requests to the AI ​​model on the node (i.e., the computing server or the virtual machine running on the computing server) via the client. The AI ​​model completes inference based on the request and sends the inference results back to the client.

[0095] It should be noted that during the inference process based on the inference request, the AI ​​model divides the inference task into multiple sub-tasks, which are handled by AI models on different nodes in the cloud environment (e.g., nodes 1 to N as shown in Figure 2). Finally, the execution results of each sub-task output by the AI ​​models on different nodes are integrated, and the inference result is obtained based on the integrated execution result.

[0096] This application provides a model inference method, wherein the model is deployed on multiple computing nodes. The computing nodes in this application can be the server shown in Figure 1, a virtual machine on the server in Figure 1, or a software module within the server or virtual machine in Figure 1; specifically, this application does not limit the computing nodes used in this embodiment.

[0097] In this embodiment, multiple computing nodes execute the model inference method provided in this embodiment based on different head Q matrices. For ease of description, this embodiment uses the first computing node among multiple computing nodes as an example for illustration. Specifically, as shown in Figure 3, the method includes: S110-S140.

[0098] S110, The first computing node receives an inference request.

[0099] The reasoning request in this application includes at least one sequence (i.e., an input sequence); wherein the sequence may be a piece of text, a photograph, or a voice sentence; specific embodiments of this application do not limit the type of the sequence.

[0100] For example, the reasoning request includes the sequence "What is AI?".

[0101] The inference request in this application can be entered by the user on the client shown in Figure 2, or it can be sent by other devices or nodes (such as management nodes). The specific embodiments of this application do not limit the source of the inference request.

[0102] It should be understood that, in response to an inference request, the first compute node runs the model deployed on the first compute node to perform the following S120-S140.

[0103] It should be noted that each of the multiple computing nodes where the first computing node is located receives the same inference request, and each computing node responds to the inference request by executing the following S120-S140.

[0104] S120, The first computing node generates the first KV matrix.

[0105] The first KV matrix in this application includes: at least one header K matrix among a plurality of header K matrices of the K matrix (key matrix) corresponding to the inference request, and at least one header V matrix among a plurality of header V matrices of the V matrix (value matrix) corresponding to the inference request.

[0106] The generation process of the first KV matrix in this application includes: generating a high-dimensional matrix sequence from the sequence requested by the inference request through the encoder in the model. Then, through the weight matrix W... K Perform a linear transformation to generate at least one head matrix K. Correspondingly, pass the sequence requested by the inference request (i.e., the sequence in high-dimensional matrix form) through the weight matrix W. V Perform a linear transformation to generate at least one head V matrix. The specific generation process is detailed in common techniques and will not be elaborated here.

[0107] It should be noted that the KV matrices generated by each of the multiple computing nodes containing the first computing node can all be the same (i.e., all are the first KV matrices); alternatively, there can be a second computing node among these multiple computing nodes that generates the same KV matrix as the first computing node, and there can also be a third computing node among these multiple computing nodes that generates a different KV matrix than the first computing node. Wherein, when the third computing node determines at least one head K matrix with the first computing node, the W it is based on... K The matrix is ​​different heads W K Matrix. Similarly, when the third computation node and the first computation node determine at least one head V matrix, the W on which it is based... V The matrix is ​​different heads W V matrix.

[0108] It should be understood that in the KV matrices determined by the same computing node, at least one head K matrix and at least one head V matrix belong to the same semantic space (i.e., head).

[0109] For example, if each row of at least one head K matrix corresponds to semantic space A and semantic space B, then each row of at least one head V matrix also corresponds to semantic space A and semantic space B.

[0110] S130, The first computing node obtains the head Q matrix set.

[0111] The head Q matrix set in this application includes: a first head Q matrix and a second head Q matrix. The first head Q matrix is ​​the head Q matrix generated by the first computation node among multiple head Q matrices; the second head Q matrix is ​​the head Q matrix generated by the second computation node among multiple head Q matrices; these multiple head Q matrices are multiple head matrices of the query matrix (Q matrix) corresponding to the inference request. That is, computation nodes that generate the same KV matrix among multiple computation nodes send their respective generated head Q matrices to each other.

[0112] It should be noted that different computing nodes among the multiple computing nodes where the first computing node resides are used to generate different head Q matrices among multiple head Q matrices. The semantic space of the first KV matrix includes: the semantic space of the first head Q matrix and the semantic space of the second head Q matrix.

[0113] For example, suppose the first head Q matrix is ​​the head Q matrix corresponding to semantic space A, and the second head Q matrix is ​​the head Q matrix corresponding to semantic space B; then the semantic space of the first KV matrix includes at least semantic space A and semantic space B.

[0114] It should be understood that the generation process of the first Q-matrix includes: passing the sequence requested by the inference request through the weight matrix W. Q A linear transformation is performed to generate the first-head Q matrix. The first computing node generates the first-head Q matrix, and the second computing node generates the second-head Q matrix, based on W. Q The matrix is ​​different heads W Q matrix.

[0115] S140. The first computing node obtains the inference result corresponding to the inference request based on the first K matrix, the first V matrix and the head Q matrix set.

[0116] In this application, the first K matrix is ​​a partial matrix of at least one head K matrix in the first KV matrix, and the first V matrix is ​​a partial matrix of at least one head V matrix in the first KV matrix.

[0117] The method for determining the first K matrix and the first V matrix in this application includes: dividing at least one head K matrix into multiple K matrices; then determining the first K matrix from the multiple K matrices. Correspondingly, dividing at least one head V matrix into multiple V matrices, and then determining the first V matrix from the multiple V matrices; for specific implementation, please refer to S240-S250 below, which will not be repeated here.

[0118] It should be noted that the first computing node and the second computing node obtain their respective inference results based on different K matrices from multiple K matrices and different V matrices from multiple V matrices.

[0119] The specific implementation of S140 in this application is described in S270-S290 below, and will not be repeated here.

[0120] The model inference method provided in this application is applied to a first computing node among multiple computing nodes. Responding to an inference request, the first computing node executes the following steps by running a model: generating a first KV matrix; wherein the first KV matrix includes at least one head K matrix of the K matrix corresponding to the inference request, and at least one head V matrix of the V matrix corresponding to the inference request. Obtaining a head Q matrix set; the head Q matrix set includes a first head Q matrix and a second head Q matrix; the first head Q matrix is ​​the head Q matrix generated by the first computing node among multiple head Q matrices, and the second head Q matrix is ​​the head Q matrix generated by the second computing node among multiple head Q matrices; the KV matrix generated by the second computing node is the first KV matrix. Then, based on a portion of the matrix in at least one head K matrix and a portion of the matrix in at least one head V matrix, and the head Q matrix set, the inference result corresponding to the inference request is obtained. It can be seen that in this application, the first computing node only needs to cache a portion of the K matrix and a portion of the V matrix in the first KV matrix, and does not need to cache the full data of the first KV matrix on the first computing node; therefore, it saves the caching resources of the computing node.

[0121] Based on Figure 3, as shown in Figure 4, this application embodiment provides a specific implementation method, which includes: S210-S290.

[0122] S210, The first computing node receives an inference request.

[0123] S220, The first computing node generates the first KV matrix and the first head Q matrix.

[0124] It should be noted that the implementation of S210-S220 is similar to that of S110-S120. For a detailed description of S210-S220, please refer to the above description of S110-S120. It will not be repeated here.

[0125] S230. The first computing node divides the first KV matrix into a target number of K matrices and a target number of V matrices based on a preset partitioning strategy.

[0126] The preset segmentation strategies in this application include: segmentation according to the batch dimension and / or segmentation according to the sequence length dimension.

[0127] The target quantity in this application is an integer greater than or equal to 2.

[0128] When the preset segmentation strategy includes segmentation according to the batch dimension, the implementation method of S230 includes: segmenting at least one head K matrix in the first KV matrix into a target number of K matrices according to the batch dimension; and segmenting at least one head V matrix in the first KV matrix into a target number of V matrices according to the batch dimension.

[0129] For example, taking at least one head K matrix in the first KV matrix as an example, assuming that at least one head K matrix is ​​the target head K matrix, as shown in Figure 5, the target head K matrix includes: batch 1 head K matrix, batch 2 head K matrix, batch 3 head K matrix, and batch 4 head K matrix; further assuming that the target quantity is 2. Then, the target head K matrix is ​​divided into 2 K matrices according to the batch dimension. The first K matrix includes: batch 1 head K matrix and batch 2 head K matrix; the second K matrix includes: batch 3 head K matrix and batch 4 head K matrix.

[0130] When the preset segmentation strategy includes segmentation according to the sequence length dimension, the implementation method of S230 includes: segmenting at least one head K matrix in the first KV matrix into a target number of K matrices according to the sequence length dimension; and segmenting at least one head V matrix in the first KV matrix into a target number of V matrices according to the sequence length dimension.

[0131] For example, taking at least one head K matrix in the first KV matrix as an example, assume that at least one head K matrix is ​​the target head K matrix. As shown in Figure 6, the sequence length corresponding to the target head K matrix is ​​6; further assume that the number of targets is 2. Then, the target head K matrix is ​​divided into 2 K matrices; the first K matrix includes: the matrix of the target head K matrix in the interval of sequence length 1 to sequence length 3; the second K matrix includes: the matrix of the target head K matrix in the interval of sequence length 4 to sequence length 6.

[0132] When the preset segmentation strategy includes segmentation according to the batch dimension and segmentation according to the sequence length dimension, the implementation method of S230 includes: segmenting at least one head K matrix in the first KV matrix into a target number of K matrices according to the batch dimension and the sequence length dimension; and segmenting at least one head V matrix in the first KV matrix into a target number of V matrices according to the batch dimension and the sequence length dimension.

[0133] For example, taking at least one head K matrix in the first KV matrix as an example, assume that at least one head K matrix is ​​the target head K matrix. As shown in Figure 7, the target head K matrix includes head K matrices of 4 batches, and the sequence length corresponding to the target head K matrix is ​​6; further assume that the target quantity is 4. Then, the target head K matrix is ​​divided into 4 K matrices; the first K matrix includes: matrices of batches 1 and 2, and sequence lengths ranging from 1 to 3; the second K matrix includes: matrices of batches 3 and 4, and sequence lengths ranging from 1 to 3; the third K matrix includes: matrices of batches 1 and 2, and sequence lengths ranging from 4 to 6; the fourth K matrix includes: matrices of batches 3 and 4, and sequence lengths ranging from 4 to 6.

[0134] In one example, the target data in this application can specifically be: the number of computing nodes among the multiple computing nodes where the first computing node resides that generate the same KV matrix; that is, the target data is the number of computing nodes that generate the same KV matrix; in other words, the target data is the sum of the number of the second computing nodes and 1 (i.e., the first computing node). This allows different computing nodes to determine the inference result based on the target number of K matrices and the target number of V matrices, with different K matrices and different V matrices; it solves the problem of redundant storage of KV matrices on different computing nodes, thus improving the cache utilization rate in computing nodes.

[0135] This embodiment of the application divides the first KV matrix into a target number of K matrices and a target number of V matrices based on a preset segmentation strategy that segments according to the batch dimension and / or according to the sequence length dimension. Since the target number of K matrices includes the first K matrix and the target number of V matrices includes the first V matrix, and the inference result is subsequently obtained based on the first K matrix and the first V matrix, the first computing node does not need to cache the first KV matrix; it only needs to cache the first K matrix and the first V matrix. This reduces the amount of data in the first KV matrix cached on the first computing node, thus saving cache resources on the first computing node.

[0136] S240. The first computing node determines the first K matrix from the target number of K matrices based on the identifier of the first computing node.

[0137] It should be noted that in this application, the multiple computing nodes that generate the first KV matrix (e.g., the first computing node and the second computing node) determine different K matrices (e.g., the first K matrix) from the target number of K matrices for executing S270.

[0138] The implementation of S240 in this application includes: sorting the target number of K matrices according to a first preset order. Then, sorting the identifiers of the first computing node and the identifiers of the second computing node according to a second preset order (e.g., sorting by size or string length). Finally, taking the K matrix corresponding to the target permutation index in the sorted target number of K matrices as the first K matrix; wherein, the target permutation index is the permutation index of the first computing node among the sorted first computing node and second computing node.

[0139] For example, suppose the target number of K matrices include K matrix A and K matrix B; the sorted order of K matrix A and K matrix B is: K matrix B, K matrix A. Further suppose that the number of computation nodes generating the first KV matrix is ​​2; the sorted order of the first and second computation nodes is: first computation node, second computation node. Since the first computation node has a sequence number of 1 in the sorted first and second computation nodes, K matrix B, with a sequence number of 1 in the sorted K matrix A and K matrix B (i.e., K matrix B, K matrix A), is determined as the first K matrix.

[0140] It should be understood that the first computing node also includes obtaining the identifier of the first computing node before executing S240.

[0141] S250. The first computing node determines the first V matrix from the target number of V matrices based on the identifier of the first computing node.

[0142] It should be noted that in this application, the multiple computing nodes that generate the first KV matrix (e.g., the first computing node and the second computing node) determine the V matrix (e.g., the first K matrix) used to execute S270 from the target number of V matrices.

[0143] The implementation of S250 in this application is similar to that of S240. For a detailed description of S250, please refer to the above description of S240. It will not be repeated here.

[0144] It should be noted that, when the preset segmentation strategy includes segmentation by batch dimension, the first K matrix and the first V matrix belong to the same batch. When the preset segmentation strategy includes segmentation by sequence length dimension, the first K matrix and the first V matrix belong to the same position interval of the sequence (the length interval from sequence length 1 to sequence length 2). When the preset segmentation strategy includes both segmentation by batch dimension and segmentation by sequence length dimension, the first K matrix and the first V matrix belong to the same batch and the same position interval of the sequence.

[0145] In this application, the first computing node may execute S240 first and then S250; or it may execute S250 first and then S240; or it may execute S240 and S250 simultaneously; specifically, the execution order of S240 and S250 is not limited in the embodiments of this application.

[0146] Based on the identifier of the first computing node, the first K matrix and the first V matrix are determined from the target number of K matrices and the target number of V matrices, respectively. This ensures that multiple computing nodes that generate the same KV matrix can obtain their own inference results based on different K matrices and V matrices, thus improving the accuracy of the inference results.

[0147] Optionally, after determining the first K matrix, the method further includes: the first computing node deleting at least one matrix other than the first K matrix from the head K matrix (i.e., the target number of K matrices); correspondingly, the first computing node deleting at least one matrix other than the first V matrix from the head V matrix (i.e., the target number of V matrices). This reduces the amount of data in the first KV matrix cached on the first computing node, thus saving cache resources on the first computing node.

[0148] It should be understood that S240-S250 in this application is an implementation of determining the first K matrix from the target number of K matrices and determining the first V matrix from the target number of V matrices.

[0149] Optionally, determining the first K matrix from the target number of K matrices and the first V matrix from the target number of V matrices can also include: the first computing node can determine the first K matrix from the target number of K matrices and the first V matrix from the target number of V matrices by interacting with the second computing node; specifically as follows:

[0150] The first computing node obtains the identifiers of the second K matrix and the second V matrix sent by the second computing node. The second K matrix is ​​the K matrix determined by the second computing node from a target number of K matrices for executing S270; the second V matrix is ​​the V matrix determined by the second computing node from a target number of V matrices for executing S270. Then, the first computing node determines the matrices other than the second K matrix from the target number of K matrices as the first K matrix. The first computing node determines the matrices other than the second V matrix from the target number of V matrices as the first V matrix.

[0151] For example, taking the first K matrix as an example, assume that the target number of K matrices include K matrix A and K matrix B; and that there are two computing nodes (i.e., the first computing node and the second computing node) that generate the first K matrix. Then, the second computing node randomly selects one K matrix (e.g., K matrix A) from the target number of K matrices as the K matrix for executing S270, and sends the identifier of K matrix A to the first computing node. Then, the first computing node uses K matrix A and K matrix B (excluding K matrix A) as the first K matrix. The first V matrix can also be obtained in a similar manner.

[0152] S260, The first computing node obtains the second head Q matrix from the second computing node.

[0153] In this application, S260 can be implemented in the following ways: the first computing node sends an acquisition request to the second computing node, and the second computing node responds to the acquisition request by sending the second head Q matrix generated by the second computing node to the first computing node; alternatively, the second computing node can actively send the second head Q matrix generated by the second computing node to the first computing node. The specific implementation of S260 in this application does not limit the specific implementation method of S260.

[0154] It should be noted that when the preset segmentation strategy in S230 includes segmentation by batch dimension, the implementation of S260 is described in S330 below, and will not be repeated here. When the preset segmentation strategy includes segmentation by sequence length dimension, the implementation of S260 is described in S430, and will not be repeated here. When the preset segmentation strategy includes segmentation by both batch dimension and sequence length dimension, the implementation of S260 is S530-S540, and will not be repeated here.

[0155] S270. The first computing node determines the first attention matrix based on the first K matrix, the first V matrix, and the head Q matrix set.

[0156] In this application, the first attention matrix determined in S270 differs depending on the preset segmentation strategy in S230. Specifically, when the preset segmentation strategy includes segmentation according to the batch dimension, the first attention matrix is ​​as shown in method a below; when the preset segmentation strategy includes segmentation according to the sequence length dimension, the first attention matrix is ​​as shown in method b below.

[0157] Method a: The first attention matrix includes: the first sub-Q attention matrix and the second sub-Q attention matrix.

[0158] The first sub-Q attention matrix is ​​the attention matrix corresponding to the first batch of head Q matrices; the first batch of head Q matrices are matrices in the first head Q matrix that belong to the same batch as the first K matrix and the first V matrix.

[0159] The second sub-Q attention matrix is ​​the attention matrix corresponding to the second batch head Q matrix; the second batch head Q matrix is ​​the matrix in the second head Q matrix that belongs to the same batch as the first K matrix and the first V matrix.

[0160] Method b: The first attention matrix includes: the first sub-KV attention matrix and the second sub-KV attention matrix.

[0161] The first sub-KV attention matrix is ​​the attention matrix corresponding to (i.e., determined) the first head Q matrix, the first K matrix, and the first V matrix.

[0162] The second sub-KV attention matrix is ​​the attention matrix corresponding to the second head Q matrix, the first K matrix, and the first V matrix.

[0163] In one example, the implementation of S270 in this application can be based on the first K matrix, the first V matrix and the head Q matrix set, and the first attention matrix can be determined by the following formula (1).

[0164] In this application, Attention(Qi,Ki,Vi) represents the first attention matrix; where Qi identifies any matrix in the first head Q matrix and the second head Q matrix set; Ki represents the first K matrix, and Vi represents the first V matrix. T This represents the transpose of the first K-matrix. The scaling factor is indicated by `softmax()`; `softmax()` represents the normalization function.

[0165] In this application, when Qi is the first head Q matrix, Attention(Qi,Ki,Vi) represents the attention matrix corresponding to the first head Q matrix; when Qi is the second head Q matrix, Attention(Qi,Ki,Vi) represents the attention matrix corresponding to the second head Q matrix.

[0166] S280. The first computing node obtains the target attention matrix based on the first attention matrix.

[0167] The target attention matrix in this application is the attention matrix corresponding to the first head Q matrix and the first KV matrix.

[0168] In this application, the implementation of S280 differs depending on the preset segmentation strategy in S230. Specifically:

[0169] When the preset segmentation strategy includes segmentation according to the batch dimension, the implementation of S280 is described in S350 below, and will not be repeated here.

[0170] When the preset segmentation strategy includes segmentation according to the sequence length dimension, the implementation of S280 is described in S450-S460 below, and will not be repeated here.

[0171] When the preset segmentation strategy includes segmentation by batch dimension and segmentation by sequence length dimension, the implementation of S280 is described in S560-S570 below, and will not be repeated here.

[0172] S290. The first computing node obtains the inference result based on the target attention matrix.

[0173] In this application, S290 is implemented by transforming the target attention matrix determined by multiple computing nodes through a linear transformation W. O The full attention matrix is ​​obtained by fusion; then, the inference result is generated based on the full attention matrix. For the specific implementation of S290, please refer to common techniques, which will not be elaborated here.

[0174] This embodiment of the application divides the first KV matrix into a target number of K matrices and a target number of V matrices based on a preset segmentation strategy that segments according to the batch dimension and / or the sequence length dimension. Then, based on the identifier of the first computing node, the first K matrix and the first V matrix are determined from the target number of K matrices and the target number of V matrices, respectively. Then, based on the first K matrix, the first V matrix, and the head Q matrix set, a first attention matrix is ​​determined; subsequently, the first computing node obtains the target attention matrix based on the first attention matrix, and obtains the inference result based on the target attention matrix. Since the first K matrix is ​​a partial matrix in at least one head K matrix, and the first V matrix is ​​a partial matrix in at least one head V matrix, it is not necessary to cache the full data of the first KV matrix on the first computing node; therefore, the caching resources of the computing node are saved.

[0175] It should be noted that, under different preset segmentation strategies, after the first computing node receives and responds to the inference request and runs the model deployed on the first computing node, the model determines the target attention matrix in different ways. Specifically, when the preset segmentation strategy is segmentation according to the batch dimension, the specific actions performed by the model in the first computing node are as described in Example 1 below. When the preset segmentation strategy is segmentation according to the sequence length dimension, the specific actions performed by the model in the first computing node are as described in Example 2 below. When the preset segmentation strategies are segmentation according to both the batch dimension and the sequence length dimension, the specific actions performed by the model in the first computing node are as described in Example 3 below.

[0176] Specifically, in this embodiment, a preset segmentation strategy can be determined based on the batch size of the first KV matrix and the sequence length of the first KV matrix. The details are as follows:

[0177] If the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​less than the second threshold, the preset segmentation strategy is to segment according to the batch dimension.

[0178] When the batch size of the first KV matrix is ​​less than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy is to segment according to the sequence length dimension.

[0179] When the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategies include: segmentation according to the batch dimension and segmentation according to the sequence length dimension.

[0180] If the batch size of the first KV matrix is ​​less than the first threshold and the sequence length of the first KV matrix is ​​less than the second threshold, then there is no preset segmentation strategy, and the computing nodes perform the inference process based on common techniques.

[0181] It should be noted that when the batch size of the first KV matrix is ​​equal to the first threshold, the preset segmentation strategy may include segmentation according to the batch dimension, or it may not include segmentation according to the batch dimension; specifically, this application embodiment does not limit the preset segmentation strategy when the batch size of the first KV matrix is ​​equal to the first threshold.

[0182] Accordingly, when the sequence length of the first KV matrix is ​​equal to the second threshold, the preset segmentation strategy may include segmentation according to the sequence length dimension; or it may not include segmentation according to the sequence length dimension. Specifically, this application embodiment does not limit the preset segmentation strategy when the sequence length of the first KV matrix is ​​equal to the second threshold.

[0183] Example 1

[0184] Based on Figure 4, when the preset segmentation strategy is to segment according to the batch dimension, this application embodiment provides a method for determining the attention matrix, as shown in Figure 8. The method includes: S310-S350.

[0185] S310, The first computing node generates the first KV matrix and the first head Q matrix.

[0186] It should be noted that the implementation of S310 is similar to that of S120. For a detailed description of S310, please refer to the above description of S120. It will not be repeated here.

[0187] S320. The first computing node determines the first K matrix and the first V matrix by dividing the first KV matrix according to the batch dimension.

[0188] It should be noted that the implementation of S320 is similar to that of S230-S250. For a detailed description of S320, please refer to the above descriptions of S230-S250, which will not be repeated here.

[0189] S330. The first computing node performs a batch-dimensional matrix swap on the first head Q matrix and the second head Q matrix generated by the second computing node based on the first set communication primitive, to obtain the head Q matrix set.

[0190] The head Q matrix set obtained by S330 of this application includes: the first batch of head Q matrices and the second batch of head Q matrices.

[0191] It should be understood that the rules for operations between matrices from different batches include: matrices from the same batch can be operated on; matrices from different batches cannot be operated on; that is, matrices from the same batch can be operated on. For example, matrices with batch number 1 can be operated on with matrices with batch number 1.

[0192] It should be noted that, taking the K matrix as an example, as shown in Figure 5, after S320 segmentation, the first computing node determines the first K matrix as the first K matrix by selecting at least one head K matrix with batches 1-2, and the second computing node determines the second K matrix as the first K matrix by selecting the first K matrix with batches 3-4 from the same head K matrix. In other words, the first and second computing nodes each retain at least one unbatched K matrix from the head K matrix. The first V matrix and the second V matrix can be determined in a similar manner.

[0193] Based on this, to ensure that both the first head Q matrix and the second head Q matrix are operated on with at least one complete head K matrix, the first set communication primitive in S330 of this application is used to exchange the first head Q matrix generated by the first computing node and the second head Q matrix generated by the second computing node according to the batch dimension of the first K matrix and / or the batch dimension of the first V matrix determined in S320. This allows the first computing node to send the batch head Q matrices other than the first batch head Q matrix in the first head Q matrix to the second computing node; and to receive the second batch head Q matrix in the second head Q matrix sent by the second computing node; specifically, the first set communication primitive in this application can be All-to-All.

[0194] For example, suppose the first head Q matrix generated by the first computing node includes a first submatrix with batches of 1-4 (i.e., 1 to 4); the second head Q matrix generated by the second computing node includes a second submatrix with batches of 1-4. If both the first K matrix and the first V matrix have batches of 1-2, the second K matrix and the second V matrix determined by the second computing node both have batches of 3-4. Then, the first computing node sends the submatrix with batches of 3-4 from the first submatrix (referred to as the third batch head Q matrix) to the second computing node; and receives the submatrix with batches of 1-2 from the second submatrix sent by the second computing node (i.e., the second batch head Q matrix).

[0195] S340. The first computing node determines the first attention matrix based on the first K matrix, the first V matrix, and the head Q matrix set.

[0196] The first attention matrix determined in S340 of this application includes: a first sub-Q attention matrix and a second sub-Q attention matrix. The first sub-Q attention matrix is ​​the attention matrix corresponding to the first batch of head Q matrices; the second sub-Q attention matrix is ​​the attention matrix corresponding to the second batch of head Q matrices.

[0197] It should be noted that the implementation of S340 is similar to that of S270. For a detailed description of S340, please refer to the above description of S270. It will not be repeated here.

[0198] This embodiment of the application determines the first K matrix and the first V matrix by partitioning the first KV matrix according to the batch dimension. Then, based on the first set communication primitive, a batch-dimensional matrix exchange is performed on the first head Q matrix generated by the first computing node and the first head Q matrix generated by the second computing node, so that the batch of head Q matrices included in the head Q matrix set in the first computing node is consistent with the batch of the first K matrix and the first V matrix; thereby ensuring the accuracy of the first attention matrix determined based on the first K matrix, the first V matrix and the head Q matrix set.

[0199] S350, the first computing node performs a head-dimension matrix exchange on the first attention matrix and the second attention matrix generated by the second computing node based on the first set communication primitive, to obtain the target attention matrix.

[0200] The second attention matrix in this application includes a third sub-Q attention matrix and a fourth sub-Q attention matrix. The third sub-Q attention matrix is ​​the attention matrix corresponding to the third batch of head Q matrices; the third batch of head Q matrices are the matrices of the other batches in the first batch of head Q matrices, excluding the first batch of head Q matrices. The fourth sub-Q attention matrix is ​​the attention matrix corresponding to the fourth batch of head Q matrices; the fourth batch of head Q matrices are the matrices of the other batches in the second batch of head Q matrices, excluding the second batch of head Q matrices.

[0201] The target attention matrix in this application includes: a first sub-Q attention matrix and a third sub-Q attention matrix.

[0202] It should be noted that in S330 of this application, a portion of the matrix in the first head Q (i.e., the third batch head Q matrix) is sent to the second computing node. In exchange, the first computing node receives the second batch head Q matrix from the second head Q matrix sent by the second computing node; the second computing node retains the fourth batch head Q matrix from the second head Q matrix. This results in the first computing node containing both the first and second batch head Q matrices, and the second computing node containing both the third and fourth batch head Q matrices. In other words, S330 enables the first computing node to contain both the first and second batch head Q matrices with different heads but the same batch; and enables the second computing node to contain both the third and fourth batch head Q matrices with different heads but the same batch.

[0203] After the first and second computing nodes execute S340, the first attention matrix determined by the first computing node includes: a first sub-Q attention matrix and a second sub-Q attention matrix; that is, the first attention matrix includes the first sub-Q attention matrix determined based on the first batch of head Q matrices, and the second sub-Q attention matrix determined based on the second batch of head Q matrices in the second head Q matrix. Similarly, the second attention matrix determined by the second computing node includes: a third sub-Q attention matrix determined based on the third batch of head Q matrices in the first head Q matrix, and a fourth sub-Q attention matrix determined based on the fourth batch of head Q matrices.

[0204] It can be seen that both the first attention matrix determined by the first computing node and the second attention matrix determined by the second computing node include sub-attention matrices corresponding to the Q matrices of different heads.

[0205] Based on this, the implementation of S340 in this application includes: the first computing node sending the second sub-Q attention matrix to the second computing node and receiving the third sub-Q attention matrix sent by the second computing node; so that the attention matrix in the first computing node is the attention matrix corresponding to the first head Q matrix, and similarly, so that the attention matrix in the second computing node is the attention matrix corresponding to the second head Q matrix.

[0206] This embodiment of the application, based on a first set of communication primitives, performs a batch-level matrix exchange on the first head Q matrix generated by the first computing node and the first head Q matrix generated by the second computing node. After determining the first attention matrix based on the first K matrix, the first V matrix, and the head Q matrix set, a head-level matrix exchange is performed on the first attention matrix and the second attention matrix generated by the second computing node, based on the first set of communication primitives, to obtain the target attention matrix. The second attention matrix includes a third sub-Q attention matrix and a fourth sub-Q attention matrix. The third sub-Q attention matrix corresponds to the attention matrix of the third batch of head Q matrices, which are matrices from other batches of the first head Q matrices besides the first batch. The fourth sub-Q attention matrix corresponds to the attention matrix of the fourth batch of head Q matrices, which are matrices from other batches of the second head Q matrices besides the second batch. Furthermore, the target attention matrix includes both the first sub-Q attention matrix and the third sub-Q attention matrix; therefore, the target attention matrix includes the attention matrix corresponding to the first head Q matrix. Subsequently, the first computing node obtains the inference result based on this target attention matrix, thereby improving the accuracy of the inference result.

[0207] Example 2

[0208] Based on Figure 4, when the preset segmentation strategy is to segment according to the sequence length dimension, this application embodiment provides a method for determining the attention matrix, as shown in Figure 9. The method includes: S410-S460.

[0209] S410, The first computing node generates the first KV matrix and the first head Q matrix.

[0210] It should be noted that the implementation of S410 is similar to that of S120. For a detailed description of S410, please refer to the above description of S120. It will not be repeated here.

[0211] S420. The first computing node determines the first K matrix and the first V matrix by dividing the first KV matrix according to the sequence length dimension.

[0212] It should be noted that the implementation of S420 is similar to that of S230-S250. For a detailed description of S420, please refer to the relevant descriptions of S230-S250 above, which will not be repeated here.

[0213] S430. The first computing node performs full synchronization on the first head Q matrix and the second head Q matrix generated by the second computing node based on the second set communication primitive, and obtains the head Q matrix set.

[0214] In S430 of this application, both the first computing node and the second computing node after synchronization include a head Q matrix set; wherein, the head Q matrix includes: a first head Q matrix generated by the first computing node and a second head Q matrix generated by the second computing node.

[0215] It should be noted that, taking the K matrix as an example, as shown in Figure 6, after S420 segmentation, the first computing node determines the sequence numbers (i.e., sequence lengths) 1-3 in at least one head K matrix as the first K matrix, and the second computing node determines the sequence numbers 4-6 in the at least one head K matrix as the second K matrix. In other words, the first and second computing nodes each retain K matrices with different sequence length ranges from at least one head K matrix. Similarly, the first V matrix and the second V matrix can also be determined.

[0216] Based on this, in order to ensure that both the first head Q matrix and the second head Q matrix are operated on with at least one complete head K matrix, the second set communication primitive in S430 of this application is used to synchronize the first head Q matrix to the second computing node and the second head Q matrix to the first computing node, so that both the first computing node and the second computing node include the first head Q matrix and the second head Q matrix. Specifically, the second set communication primitive can be AllGather.

[0217] S440. The first computing node determines the first attention matrix based on the first K matrix, the first V matrix, and the head Q matrix set.

[0218] The first attention matrix determined in S440 of this application includes: a first sub-KV attention matrix and a second sub-KV attention matrix.

[0219] This embodiment of the application determines the first K matrix and the first V matrix by segmenting the first KV matrix according to the sequence length dimension. Then, based on the second set communication primitive, full synchronization is performed on the first head Q matrix and the second head Q matrix generated by the second computing node. Therefore, both the synchronized first computing node and the second computing node include the first head Q matrix and the second head Q matrix. This ensures that the first head Q matrix and the second head Q matrix are each operated on with matrices from different sequence intervals of at least one head K matrix, thereby guaranteeing the accuracy of the operation results.

[0220] S450, The first computing node performs a head-dimensional matrix swap between the first attention matrix and the second attention matrix generated by the second computing node based on the third set communication primitive, to obtain the intermediate attention matrix;

[0221] In this application, the second attention matrix includes a third sub-KV attention matrix and a fourth sub-KV attention matrix. The third sub-KV attention matrix is ​​the attention matrix corresponding to the first head Q matrix, the second K matrix, and the second V matrix. The fourth sub-KV attention matrix is ​​the attention matrix corresponding to the second head Q matrix, the second K matrix, and the second V matrix. The second K matrix is ​​any matrix other than the first K matrix in at least one head K matrix, and the second V matrix is ​​any matrix other than the first V matrix in at least one head V matrix.

[0222] In this application, the intermediate attention matrix includes: a first sub-KV attention matrix and a third sub-KV attention matrix.

[0223] It should be noted that after S440, the first computing node obtains the attention matrix (i.e., the first sub-KV attention matrix) corresponding to the matrices of the first head Q matrix and the partial sequences in the first KV matrix (i.e., the first K matrix and the first V matrix), as well as the attention matrix (i.e., the second sub-KV attention matrix) corresponding to the second head Q matrix and the first K matrix and the first V matrix.

[0224] Similarly, after passing through S440, the second computing node obtains the attention matrix (i.e., the third sub-KV attention matrix) corresponding to the matrix of the first head Q matrix and the matrix of the remaining part of the first KV matrix (i.e., the second K matrix and the second V matrix), and the attention matrix (i.e., the fourth sub-KV attention matrix) corresponding to the second head Q matrix and the second K matrix and the second V matrix.

[0225] As can be seen, after passing through S440, the first and second computing nodes respectively obtain the head Q matrix set (including the first head Q matrix and the second head Q matrix) and the attention matrix corresponding to the matrix of different sequence length intervals of the first KV matrix.

[0226] Based on this, the third collective communication primitive in this application is used to perform a head-dimensional matrix exchange between the first attention matrix and the second attention matrix generated by the second computing node, so that the first computing node obtains the attention moments of the matrices with different sequence length intervals of the first head Q matrix and the first KV matrix (i.e., the first sub-KV attention matrix and the third sub-KV attention matrix), and the second computing node obtains the attention moments of the matrices with different sequence length intervals of the second head Q matrix and the first KV matrix (i.e., the second sub-KV attention matrix and the fourth sub-KV attention matrix). Specifically, the third collective communication primitive in this application can be ReduceScatter.

[0227] For example, suppose the first attention matrix obtained by the first computing node includes: the first sub-KV attention matrix corresponding to sequence numbers 1-3 in the first KV matrix and the first head Q matrix, and the second sub-KV attention matrix corresponding to sequence numbers 1-3 in the first KV matrix and the second head Q matrix. Then the second attention matrix obtained by the second computing node includes: the third sub-KV attention matrix corresponding to sequence numbers 4-6 in the first KV matrix and the first head Q matrix, and the fourth sub-KV attention matrix corresponding to sequence numbers 4-6 in the first KV matrix and the second head Q matrix. Since the first head Q matrix and the second head Q matrix are Q matrices with different heads, the first computing node sends the second sub-KV attention matrix to the second computing node, and the second computing node sends the third sub-KV attention matrix to the first computing node. This results in the intermediate attention matrices in the first computing node including the first and third sub-KV attention matrices, and the intermediate attention matrices in the second computing node including the second and fourth sub-KV attention matrices.

[0228] S460. The first computing node concatenates the attention matrices with the same batch in the first sub-KV attention matrix and the third sub-KV attention matrix to obtain the target attention matrix.

[0229] The implementation of S460 in this application involves concatenating the matrices of the same batch in the first sub-KV attention matrix and the third sub-KV attention matrix. For example, the matrices of the first batch in the first KV attention matrix are concatenated with the matrices of the first batch in the third sub-KV attention matrix.

[0230] In this embodiment, the first attention matrix determined by the first computing node and the second attention matrix generated by the second computing node are swapped at the head dimension based on a third set communication primitive. This allows the first computing node to obtain attention matrices corresponding to different sequence length intervals in the first KV matrix and the first head Q matrix (i.e., the first sub-KV attention matrix and the third sub-KV attention matrix), and the second computing node to obtain attention matrices corresponding to different sequence length intervals in the first KV matrix and the second head Q matrix (i.e., the second sub-KV attention matrix and the fourth sub-KV attention matrix). Then, the first computing node concatenates the attention matrices with the same batch in the first sub-KV attention matrix and the third sub-KV attention matrix to obtain the target attention matrix. Subsequently, the first node obtains the inference result based on this target attention matrix, thereby improving the accuracy of the inference result.

[0231] Example 3

[0232] Based on Figure 4, when the preset segmentation strategy includes segmentation according to the batch dimension and segmentation according to the sequence length dimension, this application embodiment provides a method for determining the attention matrix, as shown in Figure 10. The method includes: S510-S570.

[0233] S510, The first computing node generates the first KV matrix and the first head Q matrix.

[0234] It should be noted that the implementation of S510 is similar to that of S120. For a detailed description of S510, please refer to the relevant description of S120 above, which will not be repeated here.

[0235] S520. The first computing node determines the first K matrix and the first V matrix by dividing the first KV matrix according to the batch dimension and the sequence length dimension.

[0236] The segmentation method in S520 of this application can be as follows: the first computing node first segments the first KV matrix according to the batch dimension to obtain a first segmentation result. Then, the first segmentation result is segmented according to the sequence length dimension to obtain a final segmentation result. Alternatively, the first computing node can first segment the first KV matrix according to the sequence length dimension to obtain a second segmentation result; then, the second segmentation result is segmented according to the batch dimension to obtain a final segmentation result. Specifically, this application does not limit the order of segmentation according to the batch dimension and segmentation according to the sequence length dimension in its embodiments.

[0237] It should be noted that the implementation of S520 in this application is similar to that of S230-S250. For a detailed description of S520, please refer to the above descriptions of S230-S250, which will not be repeated here.

[0238] S530. The first computing node performs a batch-dimensional matrix swap on the first head Q matrix and the second head Q matrix generated by the second computing node based on the first set communication primitive, to obtain the first head Q matrix set.

[0239] It should be noted that the implementation of S530 is similar to that of S330. For a detailed description of S530, please refer to the above description of S330. It will not be repeated here.

[0240] S540. The first computing node performs full synchronization on the first head Q matrix set and the first head Q matrix set obtained by the second computing node based on the second set communication primitive, and obtains the head Q matrix set.

[0241] The set of head Q matrices in S540 of this application includes: first sub-Q matrices to eighth sub-Q matrices. Specifically, the first sub-Q matrix corresponds to sequence numbers 1-3 in the first batch of head Q matrices; the second sub-Q matrix corresponds to sequence numbers 4-6 in the first batch of head Q matrices; the third sub-Q matrix corresponds to sequence numbers 1-3 in the second batch of head Q matrices; the fourth sub-Q matrix corresponds to sequence numbers 4-6 in the second batch of head Q matrices; the fifth sub-Q matrix corresponds to sequence numbers 1-3 in the third batch of head Q matrices; the sixth sub-Q matrix corresponds to sequence numbers 4-6 in the third batch of head Q matrices; the seventh sub-Q matrix corresponds to sequence numbers 1-3 in the fourth batch of head Q matrices; and the eighth sub-Q matrix corresponds to sequence numbers 4-6 in the fourth batch of head Q matrices.

[0242] It should be noted that the implementation of S540 is similar to that of S430. For a detailed description of S540, please refer to the above description of S430. It will not be repeated here.

[0243] S550, the first computing node determines the third attention matrix based on the first K matrix, the first V matrix, and the head Q matrix set.

[0244] The third attention matrix in S550 of this application includes: first sub-attention matrices to eighth sub-attention matrices. Specifically, the first sub-attention matrix is ​​the attention matrix corresponding to the first sub-Q matrix, the first K matrix, and the first V matrix (abbreviated as: the attention matrix corresponding to the first sub-Q matrix). The second sub-attention matrix is ​​the attention matrix corresponding to the second sub-Q matrix; the third sub-attention matrix is ​​the attention matrix corresponding to the third sub-Q matrix; the fourth sub-attention matrix is ​​the attention matrix corresponding to the fourth sub-Q matrix; the fifth sub-attention matrix is ​​the attention matrix corresponding to the fifth sub-Q matrix; the sixth sub-attention matrix is ​​the attention matrix corresponding to the sixth sub-Q matrix; the seventh sub-attention matrix is ​​the attention matrix corresponding to the seventh sub-Q matrix; and the eighth sub-attention matrix is ​​the attention matrix corresponding to the eighth sub-Q matrix.

[0245] It should be noted that the implementation of S550 is similar to that of S270. For a detailed description of S550, please refer to the relevant description of S270 above, which will not be repeated here.

[0246] S560. The first computing node performs a head-dimension matrix swap between the third attention matrix and the fourth attention matrix generated by the second computing node based on the third set communication primitive, to obtain the first attention matrix.

[0247] The fourth attention matrix in S560 of this application includes: a ninth sub-attention matrix corresponding to the first sub-Q matrix, the second K matrix, and the second V matrix; a tenth sub-attention matrix corresponding to the second sub-Q matrix, the second K matrix, and the second V matrix; an eleventh sub-attention matrix corresponding to the third sub-Q matrix, the second K matrix, and the second V matrix; a twelfth sub-attention matrix corresponding to the fourth sub-Q matrix, the second K matrix, and the second V matrix; a thirteenth sub-attention matrix corresponding to the fifth sub-Q matrix, the second K matrix, and the second V matrix; a fourteenth sub-attention matrix corresponding to the sixth sub-Q matrix, the second K matrix, and the second V matrix; a fifteenth sub-attention matrix corresponding to the seventh sub-Q matrix, the second K matrix, and the second V matrix; and a sixteenth sub-attention matrix corresponding to the eighth sub-Q matrix, the second K matrix, and the second V matrix.

[0248] The first attention matrix includes: the attention matrix corresponding to the first batch of head Q matrices and the first KV (abbreviated as: the attention matrix corresponding to the first batch of head Q matrices), and the attention matrix corresponding to the second batch of head Q matrices.

[0249] In this application, S560 is implemented as follows: the first computing node sends the sub-Q matrices of the third batch of head Q matrices and the fourth batch of head Q matrices, along with the sub-attention matrices corresponding to the first K matrix and the first V matrix, to the second computing node. It also receives the sub-Q matrices of the first batch of head Q matrices and the second batch of head Q matrices, along with the sub-attention matrices corresponding to the second K matrix and the second V matrix, sent by the second computing node. Then, the sub-attention matrices corresponding to the sub-Q matrices of the first batch of head Q matrices are concatenated to obtain the attention matrix corresponding to the first batch of head Q matrices; similarly, the sub-attention matrices corresponding to the sub-Q matrices of the second batch of head Q matrices are concatenated to obtain the attention matrix corresponding to the second batch of head Q matrices. For specific implementation details, see S450-S460, which will not be elaborated here.

[0250] S570, the first computing node performs a head-dimension matrix exchange between the first attention matrix and the second attention set in the second computing node based on the first set communication primitive, to obtain the target attention matrix.

[0251] It should be understood that the target attention matrix in S570 of this application includes: the attention matrix corresponding to the first batch of head Q matrices and the attention matrix corresponding to the third batch of head Q matrices; that is, the target attention matrix includes the attention matrix corresponding to the first head Q matrix and the first KV matrix.

[0252] The implementation of S570 in this application is as follows: the first computing node sends the attention matrix corresponding to the second batch of head Q matrices to the second computing node, and receives the attention matrix corresponding to the third batch of head Q matrices sent by the second computing node.

[0253] In one implementation, the embodiments of this application can be based on a user-specified preset segmentation strategy. Referring to Figure 3 and Figure 11, before S130, the model inference method provided in the embodiments of this application further includes: S610-S620.

[0254] S610, the first computing node outputs the first interface.

[0255] The first interface in this application includes a splitting option and a non-splitting option.

[0256] For example, the first interface, as shown in Figure 12, includes a splitting option and a non-splitting option.

[0257] The non-segmentation option in this application is used to indicate the determination of the reasoning result based on common techniques, which will not be elaborated here.

[0258] It should be noted that the segmentation option in this application can be used to instruct the first computing node to determine the target attention matrix based on any one of Embodiments 1 to 3, and to determine the inference result based on the target attention matrix. It can also be used to instruct the first computing node to determine a preset segmentation strategy based on the batch size and sequence length of the first KV matrix; and to determine the scheme to be executed from Embodiments 1 to 3 based on the preset segmentation strategy, and to execute the scheme to be executed.

[0259] In one implementation, the segmentation options in this application include at least one of a first segmentation option, a second segmentation option, and an adaptive segmentation option.

[0260] The first segmentation option indicates that the preset segmentation strategy is to segment according to the batch dimension.

[0261] The second segmentation option indicates that the preset segmentation strategy is to segment according to the sequence length dimension.

[0262] The adaptive segmentation option is used to indicate the following:

[0263] When the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​less than the second threshold, the preset segmentation strategy includes: segmenting according to the batch dimension;

[0264] When the batch size of the first KV matrix is ​​less than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes: segmenting according to the sequence length dimension.

[0265] When the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategies include: segmentation according to the batch dimension and segmentation according to the sequence length dimension.

[0266] It should be noted that when the batch size of the first KV matrix is ​​equal to the first threshold and / or the sequence length of the first KV matrix is ​​equal to the second threshold, the preset segmentation strategy may include segmentation according to the batch dimension, segmentation according to the sequence length dimension, or segmentation according to both the batch dimension and the sequence length dimension.

[0267] For example, as shown in Figure 13, the first interface includes: no segmentation option, first segmentation option, second segmentation option, and adaptive segmentation option.

[0268] The segmentation options in the first interface of this application embodiment include at least one of a first segmentation option, a second segmentation option, and an adaptive segmentation option. The first segmentation option indicates that the preset segmentation strategy is segmentation according to the batch dimension; the second segmentation option indicates that the preset segmentation strategy is segmentation according to the sequence length dimension. The adaptive segmentation option indicates that: when the batch size of the first KV matrix is ​​greater than a first threshold and the sequence length of the first KV matrix is ​​less than a second threshold, the preset segmentation strategy includes segmentation according to the batch dimension; when the batch size of the first KV matrix is ​​less than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes segmentation according to the batch dimension and segmentation according to the sequence length dimension; when the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes both segmentation according to the batch dimension and segmentation according to the sequence length dimension. Thus, the preset segmentation strategy is determined by the user's selection in the first interface, thereby improving the flexibility of the preset segmentation strategy.

[0269] In one implementation, the executing entity of the above S610 can also be the cloud management platform in Figure 1; the cloud management platform outputs the first interface to the client so that the client can display the first interface.

[0270] S620, The first computing node obtains the target option selected by the user from the first interface.

[0271] It should be understood that S130 described above is executed on the first computing node when the target option is a splitting option. When the target option is a non-splitting option, the inference result is determined based on common techniques.

[0272] It should be noted that, when the execution entity of S610 is the cloud management platform shown in Figure 1, the implementation method of S620 includes: the first computing node receiving the target option from the cloud management platform. Specifically, after the user selects the target option on the first interface, the client in Figure 1 sends the identifier of the target option to the cloud management platform. After receiving the identifier of the target option, the cloud management platform sends the target option to multiple computing nodes.

[0273] This application embodiment improves the flexibility of executing the model reasoning method provided by this application embodiment by obtaining the target option selected by the user from the first interface and determining whether to execute the method provided by this application embodiment.

[0274] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, the model inference device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0275] This application embodiment can, according to the above method, exemplarily divide the model inference device into functional modules. For example, the model inference device may include functional units corresponding to each functional division, or two or more functions may be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0276] With each functional unit divided according to its corresponding function, Figure 14 shows a possible structural schematic diagram of the model inference device involved in the above embodiment; the model inference device includes: a transceiver module 1401, a running module 1402, a generation module 1403, and an inference module 1404.

[0277] The transceiver module 1401 is used to receive inference requests; for example, to execute step S110 in the above method embodiment.

[0278] The execution module 1402 is used to run the model in response to inference requests.

[0279] The generation module 1403 is used to generate the first KV matrix; for example, by performing step S120 in the above method embodiment.

[0280] The transceiver module 1401 is also used to obtain the head Q matrix set; for example, by performing step S130 in the above method embodiment.

[0281] The inference module 1404 is used to obtain the inference result corresponding to the inference request based on the first K matrix, the first V matrix and the head Q matrix set; for example, by executing step S140 in the above method embodiment.

[0282] Optionally, the model inference apparatus may also include a segmentation module 1405.

[0283] The segmentation module 1405 is used to segment the first KV matrix into a target number of K matrices and a target number of V matrices based on a preset segmentation strategy; for example, it executes step S230 in the above method embodiment.

[0284] Optionally, the target number is the number of computing nodes that generate the KV matrix among multiple computing nodes, which is the number of computing nodes that generate the first KV matrix.

[0285] Optionally, the transceiver module 1401 is used to obtain the identifier of the first computing node.

[0286] The inference module 1404 is used to determine the first K matrix from the target number of K matrices based on the identifier of the first computing node; for example, by performing step S240 in the above method embodiment.

[0287] The inference module 1404 is used to determine the first V matrix from the target number of V matrices based on the identifier of the first computing node; for example, by performing step S250 in the above method embodiment.

[0288] Optionally, the generation module 1403 is used to generate the first head Q matrix; for example, by performing step S310 in the above method embodiment.

[0289] The inference module 1404 is used to perform batch-dimensional matrix swapping on the first head Q matrix and the second head Q matrix generated by the second computing node based on the first set communication primitive, to obtain a head Q matrix set; for example, to execute step S330 in the above method embodiment.

[0290] Optionally, the generation module 1403 is used to generate the first head Q matrix; for example, by performing step S410 in the above method embodiment.

[0291] The inference module 1404 is used to perform full synchronization on the first head Q matrix and the second head Q matrix generated by the second computing node based on the second set communication primitive, to obtain the head Q matrix set; for example, to execute step S430 in the above method embodiment.

[0292] Optionally, the inference module 1404 is used to determine a first attention matrix based on the first K matrix, the first V matrix, and the head Q matrix set; for example, by performing step S270 in the above method embodiment.

[0293] The inference module 1404 is also used to obtain the target attention matrix based on the first attention matrix; for example, by performing step S280 in the above method embodiment.

[0294] The inference module 1404 is also used to obtain inference results based on the target attention matrix; for example, by performing step S290 in the above method embodiment.

[0295] Optionally, the inference module 1404 is used to perform a head-dimensional matrix exchange between the first attention matrix and the second attention matrix generated by the second computing node based on the first set communication primitive to obtain the target attention matrix; for example, by performing step S350 in the above method embodiment.

[0296] Optionally, the inference module 1404 is used to perform a head-dimensional matrix exchange between the first attention matrix and the second attention matrix generated by the second computing node based on the third set communication primitive to obtain an intermediate attention matrix; for example, by performing step S450 in the above method embodiment.

[0297] The inference module 1404 is also used to concatenate the attention matrices with the same batch in the first sub-KV attention matrix and the third sub-KV attention matrix to obtain the target attention matrix; for example, by performing step S460 in the above method embodiment.

[0298] Optionally, the generation module 1403 is used to generate the first interface; for example, by performing step S610 in the above method embodiment.

[0299] The transceiver module 1401 is used to obtain the target option selected by the user from the first interface; for example, to execute step S620 in the above method embodiment.

[0300] Optionally, the inference module 1404 is used to delete at least one matrix in the head K matrix other than the first K matrix; and to delete at least one matrix in the head V matrix other than the first V matrix.

[0301] The transceiver module 1401, the execution module 1402, the generation module 1403, the inference module 1404, and the segmentation module 1405 can all be implemented in software or in hardware. For example, the implementation of the inference module 1404 will be described below. The implementation methods of the transceiver module 1401, the execution module 1402, the generation module 1403, and the segmentation module 1405 can be referenced from the implementation method of the inference module 1404.

[0302] As an example of a software functional unit, the inference module 1404 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the aforementioned computing instance may be one or more. For example, the inference module 1404 may include code running on multiple hosts / virtual machines / containers.

[0303] It should be noted that the multiple hosts / virtual machines / containers used to run this code can be distributed within the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run this code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0304] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0305] As an example of a hardware functional unit, the inference module 1404 may include at least one computing device, such as a server. Alternatively, the inference module 1404 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0306] The multiple computing devices included in the inference module 1404 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the inference module 1404 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the inference module 1404 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0307] It should be noted that, in other embodiments, the inference module 1404 can be used to execute any step in the above-described model inference method, and the transceiver module 1401, the running module 1402, the generation module 1403, and the segmentation module 1405 can all be used to execute any step in the above-described model inference method. The steps implemented by the transceiver module 1401, the running module 1402, the generation module 1403, the segmentation module 1405, and the inference module 1404 can be specified as needed. By implementing different steps in the above-described model inference method through the transceiver module 1401, the running module 1402, the generation module 1403, the segmentation module 1405, and the inference module 1404, all functions of the management device can be realized.

[0308] This application also provides a computing device, as shown in FIG15, which may include: multiple processors 301, a memory 302, and a communication interface 303. The processors 301, memory 302, communication interface 303, and multiple physical network interface cards 305 can be connected to each other via a bus 304 or other means. This computing device may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memory in the computing device.

[0309] Each of the multiple processors 301 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0310] The memory 302 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or optical memory, disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer.

[0311] The memory 302 stores executable program code, which the processor 301 executes to implement the functions of the aforementioned transceiver module 1401, execution module 1402, generation module 1403, segmentation module 1405, and inference module 1404, thereby realizing the model inference method. That is, the memory 302 stores instructions for executing the model inference method.

[0312] Alternatively, the memory 302 stores executable code, which the processor 301 executes to implement the functions of the aforementioned model inference device, thereby realizing the model inference method. That is, the memory 302 stores instructions for executing the model inference method.

[0313] In one possible implementation, the memory 302 may exist independently of the processor 301. The memory 302 can be connected to the processor 301 via a bus 304 and is used to store data, instructions, or program code. When the processor 301 calls and executes the instructions or program code stored in the memory 302, it can implement the relevant steps in the model reasoning method provided in the embodiments of this application.

[0314] In another possible implementation, the memory 302 can also be integrated with the processor 301.

[0315] The communication interface 303 can be an acquisition module used to communicate with other devices or communication networks, such as Ethernet, RAN, and wireless local area networks (WLAN). The communication interface 303 can receive commands, messages, or data. The acquisition module can be a transceiver or similar device.

[0316] Optionally, the communication interface 303 can also be a transceiver circuit located within the processor 301, used to implement signal input and signal output of the processor 301, such as fault information of the first network card. The communication interface 303 can be a wired interface (port), such as a fiber distributed data interface (FDDI) or a gigabit Ethernet (GE) interface, or the communication interface 303 can also be a wireless interface.

[0317] Each of the multiple physical network interface cards (NICs) 305 includes, but is not limited to, a peripheral component interconnect (PCI) NIC, a peripheral component interconnect express (PCIe) NIC, and a universal serial bus (USB) NIC. In this application, the physical NICs are used to send and receive data for the processor 301.

[0318] Bus 304 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be categorized as an address bus, data bus, control bus, etc. Buses can also be classified as serial buses and parallel buses. For ease of representation, only one thick line is used in Figure 15, but this does not indicate that there is only one bus or one type of bus.

[0319] It should be understood that the computing device in Figure 15 is merely an example of a computing device, which may have more or fewer components than those shown in Figure 15, may combine two or more components, or may have different component configurations. For example, the computing device may also include a smart network interface card, such as a data processing unit (DPU).

[0320] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0321] As shown in Figure 16, the computing device cluster includes at least one computing device 100. The memory 302 of one or more computing devices 100 in the computing device cluster may store the same instructions for executing the model inference method described above.

[0322] In some possible implementations, each computing device 100 in the computing device cluster stores a portion of the instructions for executing the aforementioned model inference method in its memory 302. In other words, each computing device 100 independently executes the instructions for the aforementioned model inference method.

[0323] It should be understood that the processor and bus in the computing device 100 described above are the same as the processor 301 and bus 304 in FIG15. For a detailed description of the processor and bus in the computing device 100, please refer to the relevant description for FIG15, which will not be repeated here.

[0324] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0325] Figure 17 illustrates one possible implementation. As shown in Figure 17, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 302 in computing device 100A stores instructions for performing the functions of the transceiver module 1401 and the execution module 1402. Simultaneously, the memory 302 in computing device 100B stores instructions for performing the functions of the generation module 1403, the inference module 1404, and the segmentation module 1405.

[0326] It should be understood that the functions of computing device 100A shown in Figure 17 can also be performed by multiple computing devices 100. Similarly, the functions of computing device 100B can also be performed by multiple computing devices 100.

[0327] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster described in Figures 16 and 17. The difference is that the memory 302 of one or more computing devices 100 in this computing device cluster can store the same instructions for executing the model inference method.

[0328] In some possible implementations, the memory 302 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the model inference method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the model inference method.

[0329] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a model inference method.

[0330] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a model inference method.

[0331] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model inference method, comprising: The model performs inference based on a multi-head attention mechanism; The model is deployed on multiple computing nodes, and the method is applied to a first computing node among the multiple computing nodes. The method includes: Receive inference request; In response to the inference request, the model is run to perform the following steps: Generate a first KV matrix; wherein the first KV matrix includes: at least one head K matrix among the plurality of head K matrices of the K matrix corresponding to the inference request, and at least one head V matrix among the plurality of head V matrices of the V matrix corresponding to the inference request; Obtain a set of head Q matrices; wherein the set of head Q matrices includes: a first head Q matrix and a second head Q matrix; the first head Q matrix is ​​a head Q matrix generated by the first computing node among a plurality of head Q matrices, the second head Q matrix is ​​a head Q matrix generated by the second computing node among a plurality of head Q matrices, the plurality of head Q matrices are multiple head matrices of the Q matrix corresponding to the inference request; different computing nodes are used to generate different head Q matrices among the plurality of head Q matrices, and the KV matrix generated by the second computing node is the first KV matrix; Based on the first K matrix, the first V matrix, and the head Q matrix set, the inference result corresponding to the inference request is obtained; the first K matrix is ​​a partial matrix in the at least one head K matrix, and the first V matrix is ​​a partial matrix in the at least one head V matrix.

2. The method of claim 1, wherein, The method further includes: Based on a preset segmentation strategy, the first KV matrix is ​​segmented into a target number of K matrices and a target number of V matrices; The preset segmentation strategy includes: segmentation according to the batch dimension and / or segmentation according to the sequence length dimension; the target number is greater than or equal to 2; the target number of K matrices includes the first K matrix; the target number of V matrices includes the first V matrix; When the preset segmentation strategy includes segmentation according to the batch dimension, the first K matrix and the first V matrix belong to the same batch; When the preset segmentation strategy includes segmentation according to the sequence length dimension, the first K matrix and the first V matrix belong to the same position interval of the sequence.

3. The method according to claim 2, characterized in that, The target number is the number of computing nodes that generate the KV matrix from the plurality of computing nodes, and the number of computing nodes that generate the first KV matrix.

4. The method according to claim 2 or 3, characterized in that, The method further includes: Obtain the identifier of the first computing node; Based on the identifier of the first computing node, the first K matrix is ​​determined from the target number of K matrices; Based on the identifier of the first computing node, the first V matrix is ​​determined from the target number of V matrices.

5. The method according to any one of claims 2-4, characterized in that, When the preset segmentation strategy includes segmentation according to the batch dimension, the head Q matrix set specifically includes: a first batch of head Q matrices and a second batch of head Q matrices; wherein, the first batch of head Q matrices is a matrix in the first head Q matrix and belongs to the same batch as the first K matrix and the first V matrix; the second batch of head Q matrices is a matrix in the second head Q matrix and belongs to the same batch as the first K matrix and the first V matrix.

6. The method of claim 5, wherein, The acquisition of the head Q matrix set includes: Generate the first head Q matrix; Based on the first set of communication primitives, a batch-dimensional matrix swap is performed on the first head Q matrix and the second head Q matrix generated by the second computing node to obtain the head Q matrix set.

7. The method according to any one of claims 2-4, characterized in that, When the preset segmentation strategy includes segmentation according to the sequence length dimension, obtaining the head Q matrix set includes: Generate the first head Q matrix; Based on the second set communication primitive, full synchronization is performed on the first head Q matrix and the second head Q matrix generated by the second computing node to obtain the head Q matrix set; the synchronized first computing node and the second computing node both include the head Q matrix set.

8. The method according to any one of claims 2 to 7, characterized in that, The inference result corresponding to the inference request is obtained based on the first K matrix, the first V matrix, and the head Q matrix set, including: Based on the first K matrix, the first V matrix, and the head Q matrix set, a first attention matrix is ​​determined; Wherein, when the preset segmentation strategy includes segmentation according to the batch dimension, the first attention matrix includes: a first sub-Q attention matrix and a second sub-Q attention matrix; the first sub-Q attention matrix is ​​the attention matrix corresponding to the first batch of head Q matrices, and the second sub-Q attention matrix is ​​the attention matrix corresponding to the second batch of head Q matrices; the first batch of head Q matrices is a matrix in the first head Q matrix that belongs to the same batch as the first K matrix and the first V matrix; the second batch of head Q matrices is a matrix in the second head Q matrix that belongs to the same batch as the first K matrix and the first V matrix. When the preset segmentation strategy includes segmentation according to the sequence length dimension, the first attention matrix includes: a first sub-KV attention matrix and a second sub-KV attention matrix; the first sub-KV attention matrix is ​​the attention matrix corresponding to the first head Q matrix, the first K matrix, and the first V matrix, and the second sub-KV attention matrix is ​​the attention matrix corresponding to the second head Q matrix, the first K matrix, and the first V matrix; Based on the first attention matrix, a target attention matrix is ​​obtained; wherein, the target attention matrix is ​​the attention matrix corresponding to the first head Q matrix and the first KV matrix; The inference result is obtained based on the target attention matrix.

9. The method of claim 8, wherein, When the preset segmentation strategy includes segmentation according to the batch dimension, obtaining the target attention matrix based on the first attention matrix includes: Based on the first set of communication primitives, the first attention matrix and the second attention matrix generated by the second computing node are subjected to a head-dimensional matrix swap to obtain the target attention matrix; The second attention matrix includes a third sub-Q attention matrix and a fourth sub-Q attention matrix; the third sub-Q attention matrix is ​​the attention matrix corresponding to the third batch of head Q matrices, and the third batch of head Q matrices are matrices from other batches of the first head Q matrices excluding the first batch of head Q matrices; the fourth sub-Q attention matrix is ​​the attention matrix corresponding to the fourth batch of head Q matrices, and the fourth batch of head Q matrices are matrices from other batches of the second head Q matrices excluding the second batch of head Q matrices; the target attention matrix includes the first sub-Q attention matrix and the third sub-Q attention matrix.

10. The method of claim 8, wherein, When the preset segmentation strategy includes segmentation according to the sequence length dimension, obtaining the target attention matrix based on the first attention matrix includes: Based on the third set of communication primitives, the first attention matrix and the second attention matrix generated by the second computing node are swapped at the head dimension to obtain the intermediate attention matrix; The second attention matrix includes a third sub-KV attention matrix and a fourth sub-KV attention matrix; the third sub-KV attention matrix is ​​the attention matrix corresponding to the first head Q matrix, the second K matrix, and the second V matrix; the fourth sub-KV attention matrix is ​​the attention matrix corresponding to the second head Q matrix, the second K matrix, and the second V matrix; wherein the second K matrix is ​​the matrix other than the first K matrix among the at least one head K matrix, and the second V matrix is ​​the matrix other than the first V matrix among the at least one head V matrix; the intermediate attention matrix includes the first sub-KV attention matrix and the third sub-KV attention matrix. The target attention matrix is ​​obtained by concatenating the attention matrices with the same batch in the first sub-KV attention matrix and the third sub-KV attention matrix.

11. The method according to any one of claims 2-10, characterized in that, Prior to obtaining the head Q matrix set, the method further includes: Output the first interface; the first interface includes splitting options and no splitting options; Obtain the target option selected by the user from the first interface; The acquisition of the head Q matrix set includes: If the target option is the segmentation option, obtain the head Q matrix set.

12. The method according to claim 11, characterized in that, The segmentation options include at least one of: a first segmentation option, a second segmentation option, and an adaptive segmentation option; The first segmentation option is used to indicate that the preset segmentation strategy is to segment according to the batch dimension; The second segmentation option is used to indicate that the preset segmentation strategy is to segment according to the sequence length dimension; The adaptive segmentation option is used to indicate: When the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​less than the second threshold, the preset segmentation strategy includes: segmenting according to the batch dimension; When the batch size of the first KV matrix is ​​less than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes: segmenting according to the sequence length dimension; When the batch size of the first KV matrix is ​​greater than the first threshold and the sequence length of the first KV matrix is ​​greater than the second threshold, the preset segmentation strategy includes: segmentation according to the batch dimension and segmentation according to the sequence length dimension.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Delete all matrices in the at least one head K matrix except for the first K matrix; Delete all matrices in the at least one head V matrix except for the first V matrix.

14. A model inference apparatus comprising: The model performs inference based on a multi-head attention mechanism; The model is deployed on multiple computing nodes, and the model inference device is applied to the first computing node among the multiple computing nodes. The model inference device includes: a transceiver module, a running module, a generation module, and an inference module. The transceiver module is used to receive inference requests; The execution module is configured to run the model in response to the inference request to perform the following steps: The generation module is used to generate a first KV matrix; wherein the first KV matrix includes: at least one head K matrix among a plurality of head K matrices of the K matrix corresponding to the inference request, and at least one head V matrix among a plurality of head V matrices of the V matrix corresponding to the inference request. The transceiver module is further configured to acquire a head Q matrix set; wherein the head Q matrix set includes: a first head Q matrix and a second head Q matrix; the first head Q matrix is ​​a head Q matrix generated by the first computing node among a plurality of head Q matrices, the second head Q matrix is ​​a head Q matrix generated by the second computing node among a plurality of head Q matrices, and the plurality of head Q matrices are multiple head matrices of the Q matrix corresponding to the inference request; different computing nodes are used to generate different head Q matrices among the plurality of head Q matrices, and the KV matrix generated by the second computing node is the first KV matrix; The inference module is used to obtain the inference result corresponding to the inference request based on the first K matrix, the first V matrix, and the head Q matrix set; the first K matrix is ​​a partial matrix in the at least one head K matrix, and the first V matrix is ​​a partial matrix in the at least one head V matrix.

15. The apparatus of claim 14, wherein, The model inference device further includes: a segmentation module; The segmentation module is used to segment the first KV matrix into a target number of K matrices and a target number of V matrices based on a preset segmentation strategy; The preset segmentation strategy includes: segmentation according to the batch dimension and / or segmentation according to the sequence length dimension; the target number is greater than or equal to 2; the target number of K matrices includes the first K matrix; the target number of V matrices includes the first V matrix; When the preset segmentation strategy includes segmentation according to the batch dimension, the first K matrix and the first V matrix belong to the same batch; When the preset segmentation strategy includes segmentation according to the sequence length dimension, the first K matrix and the first V matrix belong to the same position interval of the sequence.

16. The apparatus of claim 15, wherein, When the preset segmentation strategy includes segmentation according to the batch dimension, the head Q matrix set specifically includes: a first batch of head Q matrices and a second batch of head Q matrices; wherein, the first batch of head Q matrices is a matrix in the first head Q matrix and belongs to the same batch as the first K matrix and the first V matrix; the second batch of head Q matrices is a matrix in the second head Q matrix and belongs to the same batch as the first K matrix and the first V matrix.

17. The apparatus according to claim 16, characterized in that, The generation module is used to generate the first head Q matrix; The inference module is used to perform a batch-dimensional matrix swap on the first head Q matrix and the second head Q matrix generated by the second computing node based on the first set communication primitive, so as to obtain the head Q matrix set.

18. A cluster of computing devices, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-13.

19. A computer-readable storage medium, characterized in that, The device stores computer instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1-13.

20. A computer program product comprising instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-13.