Method for determining output result of model, and related apparatus

Abstract

The present application belongs to the technical field of AI. Disclosed are a method for determining a output result of a model and a related apparatus. The method comprises: acquiring first input information of a network model; if the first input information hits a first intermediate operation result in a key-value cache, performing dimension-increasing processing on the first intermediate operation result by means of the network model to obtain a first key-value pair; and, on the basis of the first key-value pair, determining a first output result by means of the network model, the first output result being an output result corresponding to the first input information. The data volume of the first intermediate operation result is less than the data volume of the first key-value pair, that is, the key-value cache in the present application has a smaller data volume and occupies less storage space. Using intermediate operation results with a smaller data volume can increase the data loading speed of key-value caches and reduce the occupation of video memory by data loading, thereby accelerating reasoning and enhancing the overall operation performance of devices.

Description

Methods and related devices for determining model output results

[0001] This application claims priority to Chinese Patent Application No. 202510035114.8, filed on January 8, 2025, entitled “Method and Apparatus for Determining Model Output Results”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and related apparatus for determining the output of a model. Background Technology

[0003] Network models such as large language models (LLM) and multimodal models are used to determine the corresponding output based on user input. For example, in an LLM, the user's input is a question, and the LLM can determine the corresponding question-and-answer result based on the question. The process by which the network model determines the output can be called the network model's inference process. To reduce inference costs and improve inference efficiency, some intermediate computation results can be cached, making it easier to reuse these intermediate computation results in subsequent inference processes.

[0004] The aforementioned network models typically employ attention mechanisms, with intermediate computation results consisting of keys and values. These stored keys and values ​​are collectively referred to as a key-value cache (KV cache). However, the KV cache usually contains a large amount of data, consuming significant storage space. Excessive storage space usage can negatively impact device performance and even inference efficiency. Therefore, reducing the storage space consumption of the KV cache is a pressing issue that needs to be addressed. Summary of the Invention

[0005] This application provides a method and related apparatus for determining model output results, which can reduce the storage space occupied by KV cache, accelerate inference, and improve the overall operating performance of the device. The technical solution is as follows:

[0006] In a first aspect, a method for determining the output result of a model is provided, the method comprising: obtaining first input information of a network model; if the first input information hits a first intermediate operation result in a key-value cache, then performing dimensionality-upgrading processing on the first intermediate operation result through the network model to obtain a first key-value pair, wherein the data volume of the first intermediate operation result is less than the data volume of the first key-value pair; and determining a first output result based on the first key-value pair through the network model, wherein the first output result is the output result corresponding to the first input information.

[0007] The intermediate computation results in this application can be dimensionality-reduced results with a small data volume, which reduces the storage space occupied by the key-value cache. During inference, taking the reuse of the first intermediate computation result as an example, since the data volume of the first intermediate computation result is small, the loading speed of the first intermediate computation result is faster, which can accelerate inference and improve the overall operating performance of the device. Specifically, the first key-value pair can be obtained by performing dimensionality-up processing on the first intermediate computation result, thereby reusing the first key-value pair and improving inference efficiency.

[0008] The first intermediate operation result includes the following two cases.

[0009] In the first case, the result of the first intermediate operation is a piece of data, such as a matrix or a vector. In this case, the first intermediate operation result is subjected to a first dimensionality increase to obtain the key in the first key-value pair, and the first intermediate operation result is subjected to a second dimensionality increase to obtain the value in the first key-value pair.

[0010] In the second scenario, the first intermediate operation result includes two data sets. For example, the first intermediate operation result includes a second key-value reduction result and a second value-value reduction result. The second key-value reduction result can be a matrix or a vector, and the second value-value reduction result can also be a matrix or a vector. In this case, the second key-value reduction result is subjected to dimensionality-up processing to obtain the key in the first key-value pair, and the second value-value reduction result is subjected to dimensionality-up processing to obtain the value in the first key-value pair.

[0011] The aforementioned dimensionality enhancement process is performed by either a single-layer neural network or a deep neural network. Here, a deep neural network refers to a multi-layer neural network. Single-layer neural networks generally have higher execution efficiency, while deep neural networks offer better performance (e.g., inference accuracy).

[0012] In this application, the network model includes a first type of attention mechanism module, or includes both a first type of attention mechanism module and a second type of attention mechanism module. The first type of attention mechanism module performs dimensionality reduction on the hidden state data of the input information to obtain a dimensionality-reduced result (i.e., an intermediate computation result with a smaller data volume), and then performs dimensionality upscaling on the dimensionality-reduced result to obtain key-value pairs with a larger data volume. The second type of attention mechanism module processes the hidden state data of the input information to obtain key-value pairs. That is, the first type of attention mechanism module has the function of reducing the amount of data in the key-value cache, while the second type of attention mechanism module does not have this function.

[0013] Based on this, when the network model includes a first type of attention mechanism module and a second type of attention mechanism module, before performing dimensionality-upgrading processing on the first intermediate operation result through the network model to obtain the first key-value pair, the method further includes: obtaining first indication information, the first indication information indicating whether the first intermediate operation result was generated using the first type of attention mechanism module or the second type of attention mechanism module; the step of performing dimensionality-upgrading processing on the first intermediate operation result through the network model to obtain the first key-value pair includes: if the first indication information indicates that the first intermediate operation result was generated using the first type of attention mechanism module, then performing dimensionality-upgrading processing on the first intermediate operation result through the first type of attention mechanism module to obtain the first key-value pair.

[0014] The above describes the case where the first input information hits the key-value cache. The method further includes: if the first input information does not hit the key-value cache, then determining the hidden state data of the first input information through the network model to obtain the first hidden state data; determining the second key-value pair based on the first hidden state data through the network model; and determining the first output result based on the second key-value pair through the network model.

[0015] When the network model includes a first type of attention mechanism module and a second type of attention mechanism module, determining the second key-value pair based on the first hidden state data through the network model includes: performing dimensionality reduction processing on the first hidden state data using the first type of attention mechanism module to obtain a second intermediate operation result, and performing dimensionality increase processing on the second intermediate operation result using the first type of attention mechanism module to obtain the key-value pair of the first input information; or, processing the first hidden state data using the second type of attention mechanism module to obtain the key-value pair of the first input information. That is, either the first type of attention mechanism module or the second type of attention mechanism module can be used to perform the current inference task. Since both the first and second type of attention mechanism modules are trained modules, both can complete inference and ensure the accuracy of the inference results.

[0016] The method further includes, after performing dimensionality reduction processing on the first hidden state data using the first type of attention mechanism module to obtain the second intermediate calculation result, storing the second intermediate calculation result in the key-value cache. This reduces the storage space occupied by the key-value cache.

[0017] In some implementations, the network model includes a first type of attention mechanism module and a second type of attention mechanism module. Based on this, when the second intermediate operation result is generated by the first type of attention mechanism module, the method further includes: storing fourth indication information, which indicates that the second intermediate operation result is generated by the first type of attention mechanism module.

[0018] In one possible implementation, after processing the first hidden state data using the second type of attention mechanism module to obtain key-value pairs of the first input information, the first type of attention mechanism module can also be used to perform dimensionality reduction on the first hidden state data to obtain a second intermediate computation result, which is then stored in a key-value cache. That is, inference is performed using the second type of attention mechanism module, while the first type of attention mechanism module is also used to obtain and store a smaller amount of intermediate computation result.

[0019] Based on the above implementation, if the network model includes a first type of attention mechanism module and a second type of attention mechanism module, the method further includes: storing fourth indication information, wherein the fourth indication information indicates that the second intermediate operation result is generated using the first type of attention mechanism module.

[0020] In another possible implementation, after processing the first hidden state data through the second type of attention mechanism module to obtain the key-value pairs of the first input information, the key-value pairs of the first input information are stored in the key-value cache. That is, the key-value pairs with higher dimensionality are stored directly.

[0021] Based on the above implementation, if the network model includes a first type of attention mechanism module and a second type of attention mechanism module, the method further includes: storing fifth indication information, the fifth indication information indicating that the third intermediate operation result is generated by the second type of attention mechanism module, and the third intermediate operation result is the key-value pair of the first input information obtained by the second type of attention mechanism module.

[0022] In one possible implementation, the network model includes multiple attention mechanism modules, each corresponding to an application domain. These multiple attention mechanism modules include the first type of attention mechanism module. Based on this, before performing dimensionality reduction processing on the first hidden state data using the first type of attention mechanism module, the method further includes: determining the application domain to which the first input information belongs, thus obtaining a first application domain; and identifying the attention mechanism module corresponding to the first application domain among the multiple attention mechanism modules as the first type of attention mechanism module. That is, utilizing modules specific to an application domain to execute inference tasks within that domain can, on the one hand, improve the inference accuracy of each application domain, and on the other hand, schedule and distribute a large number of inference tasks, thereby improving inference efficiency.

[0023] In one possible implementation, when the network model includes a first type of attention mechanism module and a second type of attention mechanism module, before determining the second key-value pair based on the first hidden state data using the network model, the method further includes: obtaining second indication information, which indicates whether to use the first type of attention mechanism module or the second type of attention mechanism module for inference. For example, before the first type of attention mechanism module is trained, the second indication information indicates that the second type of attention mechanism module is used for inference; after the first type of attention mechanism module is trained, the second indication information indicates that the first type of attention mechanism module is used for inference.

[0024] As can be seen from the above, in one implementation, the result of the first intermediate operation is one piece of data; in another implementation, the result of the first intermediate operation includes two pieces of data.

[0025] Similarly, in one implementation, the second intermediate operation result is a single data point. The first hidden state data undergoes dimensionality reduction to obtain the second intermediate operation result, which includes performing a dimensionality reduction operation on the first hidden state data to obtain a second intermediate operation result containing a single data point. This implementation can be viewed as dimensionality reduction of the overall latent space of the key-value pair. It is relatively simple to implement, time-consuming, and efficient.

[0026] In another implementation, the second intermediate operation result includes two data sets, such as a first key dimensionality reduction result and a first value dimensionality reduction result. The step of performing dimensionality reduction on the first hidden state data to obtain the second intermediate operation result includes: performing a first dimensionality reduction on the first hidden state data to obtain the first key dimensionality reduction result; and performing a second dimensionality reduction on the first hidden state data to obtain the first value dimensionality reduction result. This implementation can be viewed as performing dimensionality reduction on the K-hidden space and the V-hidden space separately. Thus, different dimensionality reduction parameters (such as two sets of parameters) can be used to control whether the dimensions of the key dimensionality reduction result and the value dimensionality reduction result are the same or different after dimensionality reduction. Furthermore, since the two sets of dimensionality reduction parameters are more flexible, allowing dimensionality reduction based on the importance of K and V respectively, performance may be better.

[0027] For example, the dimensions of the first key dimensionality reduction result and the first value dimensionality reduction result are different. As an example, considering that the importance of the key (such as accuracy and precision requirements) may be higher than the importance of the value, the dimension of the first key dimensionality reduction result is higher than the dimension of the first value dimensionality reduction result.

[0028] In one possible implementation, the dimensionality reduction process described above is performed by a single-layer neural network or a deep neural network. Single-layer neural networks are generally more efficient, while deep neural networks offer better performance.

[0029] The training process for the first type of attention mechanism module will be introduced next.

[0030] In one possible implementation, before performing dimensionality reduction on the first hidden state data using the first type of attention mechanism module, the method further includes: training the first type of attention mechanism module online using the second type of attention mechanism module. That is, during the inference process using the second type of attention mechanism module, the first type of attention mechanism module is trained online using the data from the inference process of the second type of attention mechanism module. This eliminates the need for a dedicated training dataset, lowers the training threshold, and improves inference accuracy.

[0031] Taking the determination of the output result corresponding to the second input information (called the second output result) through the second type of attention mechanism module as an example, after obtaining the second input information of the network model, the hidden state data of the second input information is determined to obtain the second hidden state data. The second input information is the input information before the first input information. The second hidden state data is processed by the second type of attention mechanism module to obtain the key-value pair of the second input information (denoted as the fourth key-value pair). Based on the key-value pair of the second input information, the second output result is determined by the second type of attention mechanism module, and the first type of attention mechanism module is trained using the second hidden state data.

[0032] It should be understood that in the process of determining the second output result, the second hidden state data is also processed through the second type of attention mechanism module to obtain the query matrix corresponding to the second input information (denoted as the second query matrix). Based on the key-value pairs of the second input information and the second query matrix, the second output result is determined through the second type of attention mechanism module.

[0033] In this application, training a first-type attention mechanism module using second hidden state data includes: performing dimensionality reduction processing on the second hidden state data through the first-type attention mechanism module to obtain sample dimensionality reduction results; performing dimensionality upscaling processing on the sample dimensionality reduction results through the first-type attention mechanism module to obtain reconstructed key-value pairs; determining a loss value based on the reconstructed key-value pairs; and updating the parameters of the first attention mechanism module based on the loss value.

[0034] One implementation of dimensionality reduction processing on the second hidden state data to obtain the sample dimensionality reduction result is as follows: perform dimensionality reduction processing on the second hidden state data once to obtain a single data point, which is the sample dimensionality reduction result. Correspondingly, one implementation of dimensionality increase processing on the sample dimensionality reduction result to obtain the reconstructed key-value pairs is as follows: perform a first dimensionality increase processing on the sample dimensionality reduction result to obtain the key (denoted as the reconstructed key) in the reconstructed key-value pair; perform a second dimensionality increase processing on the sample dimensionality reduction result to obtain the value (denoted as the reconstructed value) in the reconstructed key-value pair.

[0035] Another way to reduce the dimensionality of the second hidden state data to obtain the sample dimensionality reduction result is as follows: perform a first dimensionality reduction on the second hidden state data to obtain the key sample dimensionality reduction result, and then perform a second dimensionality reduction on the second hidden state data to obtain the value sample dimensionality reduction result. That is, the sample dimensionality reduction result includes both the key sample dimensionality reduction result and the value sample dimensionality reduction result. Correspondingly, one way to increase the dimensionality of the sample dimensionality reduction result to obtain the reconstructed key-value pairs is as follows: increase the dimensionality of the key sample dimensionality reduction result to obtain the reconstructed key, and increase the dimensionality of the value sample dimensionality reduction result to obtain the reconstructed value.

[0036] In this application, there are multiple ways to determine the loss value based on the reconstruction of key-value pairs, and these multiple ways will be introduced below.

[0037] One approach to determining the loss value is to base it on the fourth key-value pair and the reconstructed key-value pair. The fourth key-value pair is determined based on the second hidden state data; for example, it can be obtained by processing the second hidden state data using a second-type attention mechanism module.

[0038] A second approach to determining the loss value is as follows: Based on the reconstructed key and the second query matrix, a reconstructed weight matrix is ​​determined. The loss value is then determined based on the second weight matrix, the reconstructed weight matrix, the values ​​in the fourth key-value pair, and the reconstructed value. The second query matrix is ​​determined based on the second hidden state data; for example, it can be obtained by processing the second hidden state data using a second-type attention mechanism module. Similarly, the second weight matrix is ​​determined based on the keys in the fourth key-value pair and the second query matrix; for instance, it can be obtained by processing the keys in the fourth key-value pair and the second query matrix using a second-type attention mechanism module.

[0039] A third approach to determining the loss value is as follows: Based on the reconstruction key and the second query matrix, determine the reconstruction weight matrix; based on the reconstruction weight matrix and the reconstructed values, determine the reconstruction output matrix; and based on the second output matrix and the reconstruction output matrix, determine the loss value. Specifically, the second query matrix is ​​determined based on the second hidden state data; the second output matrix is ​​determined based on the values ​​in the second weight matrix and the fourth key-value pair; and the second weight matrix is ​​determined based on the keys in the second key-value pair and the second query matrix.

[0040] The three methods for determining the loss value described above may have different convergence speeds and training accuracies. For example, the first method has a slower convergence speed but higher training accuracy, while the third method has a faster convergence speed but slightly lower training accuracy. The method for determining the loss value can be flexibly set according to actual needs.

[0041] As described above, the first type of attention mechanism module can correspond to the first application domain; that is, the first type of attention mechanism module is the attention mechanism module corresponding to the first application domain. Based on this, the step of using the second type of attention mechanism module to train the first type of attention mechanism module online includes: when the input information of the second type of attention mechanism module belongs to the first application domain, using the second type of attention mechanism module to train the first type of attention mechanism module online. For example, when the second input information belongs to the first application domain, using the second hidden state data to train the first type of attention mechanism module online.

[0042] As discussed above, the network model can include attention mechanism modules corresponding to multiple application domains. Any one of these attention mechanism modules can be trained using the methods described above. For example, the second type of attention mechanism module can be used for online training of any one of these attention mechanism modules. This addresses the problem of insufficient inference accuracy in scenarios where the application domains of the training data are unevenly distributed. Specifically, by training multiple attention mechanism modules, inference accuracy and precision across various application domains can be improved, while also achieving a balance in generalization.

[0043] In one possible implementation, after obtaining the current input information of the network model, the application domain to which the current input information belongs is determined, thus obtaining the target application domain. During the reasoning process using the second type of attention mechanism module on the current input information, the hidden state data of the current input information is used to train the attention mechanism module corresponding to the target application domain online. Here, the current input information refers to the input information within the online training period, which is the period during which the multiple attention mechanism modules have not yet completed training.

[0044] Secondly, an apparatus for determining model output results is provided, the apparatus having the functionality to implement the method behavior of determining model output results described in the first aspect above. The apparatus for determining model output results includes one or more modules, which are used to implement the method for determining model output results provided in the first aspect above.

[0045] Thirdly, a task execution device is provided. The network model includes a first type of attention mechanism module and a second type of attention mechanism module. The task execution device is used to perform an inference task using the second type of attention mechanism module to obtain an inference result, and to perform an online training task on the first type of attention mechanism module using the second type of attention mechanism module. The trained first type of attention mechanism module is used to perform dimensionality reduction processing on the hidden state data of the input information of the network model to obtain intermediate operation results, and to perform dimensionality increase processing on the intermediate operation results to obtain key-value pairs of the input information. The intermediate operation results are used to store in a key-value cache for reuse. That is, during the inference process of the second type of attention mechanism module, the data from the inference process of the second type of attention mechanism module is used to train the first type of attention mechanism module online.

[0046] Fourthly, a computer device is provided, comprising a processor and a memory, the memory being used to store a program for executing the method for determining the output result of a model provided in the first aspect, and to store data related to implementing the method for determining the output result of a model provided in the first aspect. The processor is configured to execute the program stored in the memory.

[0047] In one possible implementation, the computer device may further include a communication bus for establishing a connection between the processor and the memory.

[0048] Fifthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the method for determining the output result of the model as described in the first aspect.

[0049] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the method for determining the output result of the model as described in the first aspect.

[0050] The technical effects achieved by the second to sixth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0051] Figure 1 is a schematic diagram of an Attention calculation process provided in an embodiment of this application;

[0052] Figure 2 is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0053] Figure 3 is a flowchart of a method for determining model output results provided in an embodiment of this application;

[0054] Figure 4 is a schematic diagram of a deep neural network structure provided in an embodiment of this application;

[0055] Figure 5 is a flowchart of another method for determining the output result provided in an embodiment of this application;

[0056] Figure 6 is a flowchart of a method for determining key-value pairs provided in an embodiment of this application;

[0057] Figure 7 is a flowchart of another method for determining key-value pairs provided in an embodiment of this application;

[0058] Figure 8 is a flowchart of another method for determining key-value pairs provided in an embodiment of this application;

[0059] Figure 9 is a flowchart of another method for determining key-value pairs provided in an embodiment of this application;

[0060] Figure 10 is a schematic diagram of some modules in a network model provided in an embodiment of this application;

[0061] Figure 11 is a training schematic diagram of a first type of attention mechanism module provided in an embodiment of this application;

[0062] Figure 12 is a reasoning diagram of a first type of attention mechanism module provided in an embodiment of this application;

[0063] Figure 13 is a schematic diagram of a device structure for determining model output results provided in an embodiment of this application;

[0064] Figure 14 is a schematic diagram of the structure of a task execution device provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0066] To facilitate understanding, before explaining the method for determining the output result of the model provided in this application, the terminology, application scenarios, and implementation environment involved in the embodiments of this application will be introduced first.

[0067] First, the relevant terminology in the embodiments of this application will be introduced.

[0068] 1. Sequential reasoning

[0069] Sequence reasoning refers to the process of reasoning from input information (which can also be a sequence) to obtain an output sequence.

[0070] Currently, network models such as large language models, large visual models, and multimodal models can be applied to sequence reasoning. Large language models refer to a class of network models in the field of natural language processing (NLP), typically used to process text data and understand natural language, such as determining the corresponding question-and-answer result based on a user's question. The main characteristic of these network models is that they are trained on large-scale corpora to learn various grammatical, semantic, and contextual rules of natural language. Large visual models refer to network models used in the field of computer vision (CV), typically used for image processing and analysis. These models, trained on large-scale image data, can achieve various visual tasks, such as image classification, object detection, image segmentation, pose estimation, and face recognition. Multimodal models are network models capable of processing multiple different types of data, such as determining the output result of another modality based on input information in one modality provided by a user. Modality refers to the form of data; multimodal data refers to data in multiple forms. Multimodal data can include text, images, video, audio, etc. These models combine NLP and CV capabilities to achieve a comprehensive understanding and analysis of multimodal information, thereby enabling a more comprehensive understanding and processing of complex data.

[0071] One of the most popular network model architectures for sequence reasoning is the Transformer architecture. It achieves efficient processing of sequence data through an attention mechanism, yielding significant results in related fields, such as in multi-turn conversations and other long sequence reasoning applications. The attention mechanism includes several forms, such as self-attention and cross-attention. Self-attention allows the model to focus on different positions within a sequence simultaneously, capturing dependencies within the sequence; while cross-attention handles dependencies between two sequences, such as the alignment between source and target languages ​​in machine translation. The attention mechanism in the Transformer architecture calculates relevance scores between different positions in the input sequence and then performs a weighted sum based on these scores to obtain the output sequence. This computational approach allows the Transformer architecture to handle input sequences of infinite length while maintaining high efficiency and accuracy during computation.

[0072] Network models based on self-attention mechanisms in related technologies include multi-head attention (MHA) mechanism modules, multi-head latent attention (MLA) mechanism modules, motion-guided attention (MGA) mechanism modules, or DeepSeek-based MLA modules, etc.

[0073] Of course, there are also network models based on other architectures, such as models based on the Mamba architecture. In some applications, network models based on the Mamba architecture can also use self-attention mechanisms.

[0074] 2. Attention Mechanism

[0075] Referring to Figure 1, the Attention calculation process typically involves three key matrices (in this paper, these matrices can be one-dimensional vectors or multi-dimensional matrices): the Query (Q) matrix, the Key (K) matrix, and the Value (V) matrix. The dimensions of these three matrices are Q∈R. S1×d K∈R S2×d and V∈R S2×d S1 and S2 represent the lengths of the input sequences, which may be equal or unequal, and d is the feature dimension. The Attention calculation process may include the following steps:

[0076] (1) Calculate the fraction matrix.

[0077] Calculate the transpose of the query matrix (Q) T The dot product of the input sequence and the key matrix (i.e., matrix multiplication) yields a fractional matrix (denoted as S), which has dimensions S1×S2. This fractional matrix reflects the correlation between different parts of the input sequence.

[0078] The mathematical formula for calculating the fractional matrix can be: S = Q T K∈R S1×S2 .

[0079] (2) Scaling the fractional matrix.

[0080] In practical calculations, to maintain numerical stability, the fractional matrix is ​​usually scaled, i.e., divided by a constant σ.

[0081] The mathematical formula for scaling the fractional matrix is: S = S / σ∈R S1×S2 , d refers to the feature dimension described above.

[0082] (3) Apply the softmax function.

[0083] Applying the softmax function to the score matrix yields a weight matrix (also called the score matrix or probability distribution matrix, denoted as P), which still has the dimension S1×S2. This weight matrix reflects the relative importance of different parts of the input sequence.

[0084] The mathematical formula for calculating the weight matrix can be: P = softmax(S) ∈ R S1×S2 .

[0085] (4) Calculate the weighted sum to obtain the output matrix.

[0086] Performing matrix multiplication on the weight matrix P and the value matrix V yields an output matrix (denoted as O) with dimensions S1×d. This output matrix contains feature information from different parts of the input sequence weighted according to their relative importance.

[0087] The mathematical formula for calculating the output matrix can be: O = PV ∈ R S1×d .

[0088] Building upon the standard Attention computation process described above, industry professionals have also released a series of Flash Attention (FA) models. The FA operator is essentially an algorithm optimization technique that leverages the memory hierarchy of underlying hardware, such as the memory structures of central processing units (CPUs), graphics processing units (GPUs), and neural network processing units (NPUs), to improve computational speed and reduce memory access overhead. In other words, the FA operator is not a traditional hardware operator, but rather a software-level optimization method.

[0089] Compared to the standard Attention computation process described above, the FA operator employs a chunking algorithm and online softmax update. These optimizations enable the FA operator to handle theoretically unconstrained sequence lengths, while significantly improving computational performance and substantially reducing the use of high bandwidth memory (HBM).

[0090] In the standard Attention computation process, a score matrix S with dimensions S1×S2 is typically calculated, where S1 and S2 represent the lengths of the input sequence. When the sequence length is very long, this score matrix becomes extremely large, leading to a sharp increase in computational resources and memory usage. To address this, the FA operator introduces a block-based algorithm. The basic idea of ​​this algorithm is to divide the input sequence into multiple smaller chunks and then perform Attention computation on each chunk separately. This significantly reduces the computational load and memory usage within each chunk, while also allowing for parallel processing of multiple chunks using multi-level memory structures such as GPUs.

[0091] 3. Key-value cache (KV cache) technology

[0092] KV caching technology is primarily based on the self-attention mechanism in the Transformer model. In self-attention, the model first computes three datasets: query (Q), key (K), and value (V). These datasets are typically obtained by applying different linear transformations to the hidden states (embedded vectors) of the input sequence. Then, the model uses matrix multiplication to calculate the dot product of the query vector and all key vectors, generating a weight distribution. Finally, it uses a weighted average of these weights to obtain the contextual information for each position.

[0093] In the inference process of network models such as large language models and multimodal models, only one new character / word (token) is generated each time. This new character / word is then concatenated with the already generated part into the input sequence to form a longer sequence, which is then fed back into the model to generate the next token. This autoregressive characteristic means that the model needs to repeatedly process the same or similar input sequence fragments, especially in multi-turn dialogues or long text generation scenarios. Since the model has to recalculate the K and V of the entire input sequence each time it infers, and these K and V are invariant for the same input token, this results in a large amount of redundant computation.

[0094] Based on this, the KV cache technology caches previously calculated K and V vectors, allowing the model to choose to reuse the key-value cache in subsequent inference steps, avoiding redundant calculations, significantly reducing the amount of computation required, and thus speeding up inference.

[0095] In scenarios such as text generation, question answering systems, and dialogue systems, KV cache can significantly improve the response speed of models and reduce computational costs.

[0096] Secondly, the application scenarios of the embodiments of this application will be introduced.

[0097] The inference process of the network models described above includes a prefill stage and a decoding stage. The prefill stage is also called the full inference stage, and the decoding stage is also called the incremental inference stage. The prefill stage is used to perform calculations on the input information to obtain intermediate calculation results corresponding to the input information. The decoding stage is used to determine the output result based on the intermediate calculation results corresponding to the input information.

[0098] The prefill phase is a crucial preprocessing step in the large model inference process, primarily aimed at preparing for subsequent incremental inference phases. During this phase, the model loads and processes user input data, along with system-attached templates, to generate necessary intermediate results (such as key-value caches). Taking key-value cache generation as an example, in a transformer architecture model, the result of the current token is typically only related to previous tokens. Therefore, in the prefill phase, the large model computes all user inputs and generates corresponding key-value caches. These caches can be reused in subsequent incremental inference phases to reduce computational overhead.

[0099] As an example, in a text generation task, the model calculates and generates the key-value value corresponding to each token based on the text sequence input by the user, and stores these values ​​in a cache.

[0100] The decoding phase is the core of the large model inference process, and its main task is to generate tokens in the output sequence one by one. In this phase, the model gradually constructs the complete output sequence based on the KV cache generated in the prefill phase and the already generated tokens. The generation of each token is based on the previous token and the current context information. To reduce computation, the decoding phase reuses the KV cache generated in the prefill phase. During the generation of each token, the model only calculates the K and V corresponding to the current token and merges it with the key-value pair of the previous token in the cache. In this way, the model avoids repeatedly calculating tokens that have already been calculated, thereby improving inference speed.

[0101] As an example, in a text generation task, the model generates the next character step by step based on the already generated text and the currently input character, until a complete sentence or paragraph is generated. The decoding stage is an iterative generation process. In each iteration step, the model generates a new token and adds it to the output sequence. When a preset termination condition is met (such as generating text of a specific length, encountering a terminator, etc.), the network model stops generating and returns the final output sequence to the user or system.

[0102] Let's illustrate the two stages with another example. For instance, a user asks the question "What color is an apple?" The question is divided into multiple input tokens: "apple," "yes," "what," and "color." In the pre-filling stage, the intermediate computation results corresponding to each of these input tokens are determined. In the decoding stage, the first decoding block determines the output token as "apple" based on the intermediate computation results of these input tokens. The second decoding block determines the output token as "of" based on the intermediate computation results of these input tokens and the output token "apple." The third decoding block determines the output token as "color" based on the intermediate computation results of these input tokens and the output tokens "apple," "of," and "color." The fourth decoding block determines the output token as "yes" based on the intermediate computation results of these input tokens and the output tokens "apple," "of," "color," and "yes." The fifth decoding block determines the output token as "red" based on the intermediate computation results of these input tokens and the output tokens "apple," "of," "color," and "yes." Combining these output tokens yields the question-and-answer result: "The color of an apple is red."

[0103] As can be seen, the intermediate computation results corresponding to each input token determined in the pre-filling stage are used by each decoding block in the decoding stage to generate output tokens. Simultaneously, the intermediate computation results corresponding to each output token are also used by subsequent decoding blocks to generate subsequent output tokens. That is, a series of intermediate computation results are generated during the inference process, and these intermediate computation results are used multiple times during the inference process.

[0104] The data volume of intermediate calculation results (i.e., KV cache) is usually very large. To reduce the storage space occupied by the KV cache, this application provides a method for determining the model output results. The dimension of the KV cache stored by this method is lower than that of the KV cache in related technologies, that is, the data volume is smaller. The specific implementation method will be described later.

[0105] Finally, the implementation environment involved in the embodiments of this application will be described.

[0106] The method for determining the model output results provided in this application embodiment can run on a single server or server cluster, or on a terminal device or other device, and this application embodiment does not limit it in this regard.

[0107] If the method runs on a single device (such as a single server or terminal device), the device may include an inference device and a storage device. A communication connection is established between the inference device and the storage device. The storage device is used to store intermediate computation results generated during the inference process. The inference device is used to perform inference according to the method provided in the embodiments of this application, including retrieving KV cache from the storage device and reusing it.

[0108] In some implementations, the network model deployed by the inference device includes multiple inference modules, each executing different inference tasks. Based on this, the device also includes a controller that establishes a communication connection with the inference device. The controller is used to schedule and distribute inference tasks to distribute the load across the multiple inference modules. In some embodiments, the controller also establishes a communication connection with a storage device and is used to manage and maintain the data stored in the storage device. The controller can be a CPU or other device, and this application embodiment does not limit this.

[0109] If the method operates on a server cluster, the server cluster may include an inference cluster (or a single inference device) and a storage cluster (or a single storage device), with a communication connection established between the inference cluster and the storage cluster. The storage cluster is used to store intermediate computation results generated during the inference process, and the inference cluster is used to perform inference according to the method provided in the embodiments of this application, including retrieving and reusing the KV cache from the storage cluster.

[0110] In some implementations, the server cluster may further include a control center, which establishes a communication connection with the inference cluster. The control center is used to schedule and distribute inference tasks to distribute the load of inference tasks through the inference cluster. In some embodiments, the control center may also establish a communication connection with the storage cluster, and the control center is also used to manage and maintain the storage cluster and its stored data. The control center may be a separate server or integrated into a certain inference server or storage server; this application embodiment does not limit this.

[0111] In some implementations, the method can run on AI computing devices such as NPUs, GPUs, TPUs, or CPUs, which are the aforementioned devices (e.g., servers or terminal devices).

[0112] Based on this, embodiments of this application also provide a computer device. This computer device can be a single terminal device or server, or other devices, clusters, data centers, etc., including hardware such as processors, used to execute the method for determining model output results provided in embodiments of this application. When implementing the method for determining model output results provided in embodiments of this application, the processor can include, but is not limited to, CPUs, GPUs, NPUs, TPUs, etc., and embodiments of this application do not impose such limitations.

[0113] Please refer to Figure 2, which is a schematic diagram of a computer device according to an embodiment of this application. The computer device may be a server or terminal device as described above, or it may be a device in a server cluster. The computer device includes at least one processor 201, a communication bus 202, a memory 203, and at least one communication interface 204.

[0114] The processor 201 can be a general-purpose CPU, GPU, NPU, TPU, network processor (NP), microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0115] The communication bus 202 is used to transmit information between the aforementioned components. The communication bus 202 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in Figure 2, but this does not mean that there is only one bus or one type of bus.

[0116] The memory 203 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile optical disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 203 may exist independently and be connected to the processor 201 via the communication bus 202. The memory 203 may also be integrated with the processor 201.

[0117] Communication interface 204 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 204 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0118] As an example, processor 201 may include one or more CPUs, such as CPU0 and CPU1 as shown in FIG2.

[0119] As an example, a computer device may include multiple processors, such as processor 201 and processor 205 as shown in Figure 2. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0120] In some embodiments, the computer device may further include output devices and input devices. The output device communicates with the processor 201 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 201 and can receive user input in various ways. For example, the input device may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0121] In some embodiments, memory 203 is used to store program code 210 for executing the scheme of this application, and processor 201 can execute program code 210 stored in memory 203. The program code 210 may include one or more software modules, and the computer device can implement the method for determining the model output result provided in the embodiment of FIG3 below through processor 201 and program code 210 in memory 203.

[0122] It should be understood that the application scenarios, implementation environments, and computer device architectures described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of computer device architecture, the emergence of new business scenarios, and the evolution of implementation environments, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0123] The method for determining the model output results provided in the embodiments of this application will be explained in detail below.

[0124] Figure 3 is a flowchart of a method for determining model output results provided in an embodiment of this application. This method is applied in the implementation environment described above. The executing entity of this method can be a computer device or any other device. The computer device can be the computer device shown in Figure 2, which has part or all of the network model deployed on it. Taking the executing entity of this method as a computer device as an example, please refer to Figure 3. The method includes the following steps.

[0125] Step 301: Obtain the first input information of the network model.

[0126] The first input information of the network model is the input information for this model inference process. The first input information can be the input information provided by the user. That is, the user terminal obtains the input information provided by the user and sends a request to the first data center. This request carries the input information provided by the user and uses the input information provided by the user as the current input information for generating the model.

[0127] There are various ways for a user terminal to obtain user-provided input information. For example, the user terminal displays an information retrieval interface, retrieves the information provided by the user on the information retrieval interface, and uses the information provided by the user on the information retrieval interface as the user-provided input information. The form of user-provided input information can include various forms, such as text, voice, images, and video.

[0128] The first input information can also be input information provided by the device, such as input information generated by the aforementioned computer device, or input information sent by other devices. Therefore, in one possible application scenario, the first input information is automatically entered information.

[0129] In this embodiment, the network model can be any model used for sequence reasoning, such as a large language model or a multimodal model. The network model can be a large model or a small model; this embodiment does not limit this. Specifically, the network model is an attention-based model, meaning it includes an attention mechanism module. This embodiment does not limit the type and structure of the attention mechanism module. Of course, the network model can also be a model with other mechanisms, as long as it can be used for sequence reasoning. The following description uses the attention mechanism module as an example. In one implementation, the network model includes a multi-layer network structure, where some or all of the network structures in the multi-layer network structure include an attention mechanism module. These multi-layer network structures can be connected in series; specific implementation methods can refer to related technologies.

[0130] This network model includes an attention mechanism module, an input processing module, and an output processing module. The input processing module processes the input information to obtain hidden states. The attention mechanism module processes the hidden states to obtain the output matrix. The output processing module processes the output matrices of attention mechanism modules other than the last layer to obtain the input data for the next layer (i.e., the hidden states used to input the next layer), or processes the output matrix of the last layer to obtain the output result.

[0131] The input processing module can be a multi-layer neural network, a single-layer neural network, or other network structures; this application embodiment does not limit this. The output processing module can be a multi-layer perceptron (MLP) or other network structures; this application embodiment does not limit this.

[0132] Step 302: If the first input information hits the first intermediate operation result in the key-value cache, then the first intermediate operation result is upgraded through the network model to obtain the first key-value pair. The data size of the first intermediate operation result is smaller than the data size of the first key-value pair.

[0133] The key-value cache stores intermediate computation results obtained by the network model from processing historical input information. Each intermediate computation result in the key-value cache has an index. The index of the intermediate computation result can be constructed based on historical input information, such as using historical input information (i.e., historical tokens) as the index. The index can also be the number corresponding to the historical token, or other forms, as long as it can correspond to the historical token. This application embodiment does not limit this.

[0134] In this embodiment of the application, after the network model obtains the first input information, it first determines whether the first input information can hit the intermediate calculation result in the key-value cache. If it can hit, the network model does not need to recalculate the intermediate calculation result corresponding to the first input information.

[0135] The network model segments the first input information into multiple tokens, and then retrieves the indexes of intermediate operation results from the key-value cache based on these tokens. If the first input information is text, the multiple tokens represent multiple words / characters; if the first input information is an image, the multiple tokens represent multiple image blocks; if the first input information is video, the multiple tokens can represent multiple video segments or multiple image blocks. This application does not limit the segmentation method (also known as block segmentation method) for text, images, and videos.

[0136] In one implementation, to improve retrieval accuracy, the index of intermediate operation results in the key-value cache can be retrieved based on the multiple tokens and their dependencies in the first input information, using prefix matching.

[0137] As an example, the first input information is "What is your name?". The network model segments the first input information into multiple tokens, including "you", "name", "what", and "name". First, the system searches for "you" to determine if it matches a specific index. If "you" does not match, the system determines that the first input information did not hit the key-value cache. If "you" matches, the system continues searching for "your name" to determine if it matches a specific index. If "your name" does not match, the system determines that the first input information matches the intermediate operation result corresponding to the historical input information "you". If "your name" matches, the system continues searching for "what your name is" to determine if it matches a specific index. If "what your name is" does not match, the system determines that the first input information matches the intermediate operation result corresponding to the historical input information "your name". If "what your name is" matches, the system continues searching for "what your name is" to determine if it matches a specific index. If "what your name is" does not match, the system determines that the first input information matches the intermediate operation result corresponding to the historical input information "what your name is". If "what your name is" matches the name, the system determines that the first input information matches the intermediate operation result corresponding to the historical input information "what your name is".

[0138] As can be seen from the above, in the embodiments of this application, the first input information hitting the key value cache means that part or all of the first input information hits the key value cache, and the first input information not hitting the key value cache means that all of the first input information does not hit the key value cache.

[0139] If the first input information matches the first intermediate operation result in the key-value cache, the first intermediate operation result is then subjected to dimensionality upscaling processing through the network model to obtain the first key-value pair. The data size of the first intermediate operation result is smaller than the data size of the first key-value pair. In other words, to reduce the storage space occupied by the key-value cache, the intermediate operation results in the key-value cache can be dimensionality-reduced results. Dimensionality-reduced results have a smaller data size, and when reusing intermediate operation results, the dimensionality-reduced results can be upscaled to recover the key-value pair.

[0140] The first intermediate calculation result in this application embodiment has two possibilities.

[0141] In the first case, the result of the first intermediate operation is a piece of data, such as a matrix or a vector. In this case, the first intermediate operation result is subjected to a first dimensionality increase to obtain the key in the first key-value pair, and the first intermediate operation result is subjected to a second dimensionality increase to obtain the value in the first key-value pair.

[0142] For example, the network model includes a dimensionality-upgrading unit, which performs dimensionality-upgrading processing on the first intermediate computation result. In one implementation, the network model includes a first-type attention mechanism module, which includes the dimensionality-upgrading unit.

[0143] In the second scenario, the first intermediate operation result includes two data sets: a second key-value pair dimensionality reduction result and a second value-value pair dimensionality reduction result. The second key-value pair dimensionality reduction result can be either a matrix or a vector. In this case, the second key-value pair dimensionality reduction result is upgraded to obtain the key, and the second value-value pair dimensionality reduction result is upgraded to obtain the value.

[0144] For example, the network model includes a first dimensionality-upgrading unit and a second dimensionality-upgrading unit. The first dimensionality-upgrading unit performs dimensionality-upgrading on the second key dimensionality-downgrading result, and the second dimensionality-upgrading unit performs dimensionality-upgrading on the second value dimensionality-downgrading result. In one implementation, the network model includes a first type of attention mechanism module, which includes a first dimensionality-upgrading unit and a second dimensionality-upgrading unit.

[0145] The key in the first key-value pair can be a key matrix or a key vector. For ease of distinction and description, the key in the first key-value pair can be called the first key. The value in the first key-value pair can be a value matrix or a value vector. For ease of distinction and description, the value in the first key-value pair can be called the first value.

[0146] The aforementioned dimensionality enhancement process is performed by a single-layer neural network or a deep neural network. Here, a deep neural network refers to a multi-layer neural network. This application does not limit the specific structure of the single-layer neural network or the deep neural network, nor does it limit the number of layers in the deep neural network. As an example, the single-layer neural network can be a low-rank adaptation of large models (LoRA) network, and the deep neural network model can be an autoencoder (AE) network or an AE-like network, such as a variational autoencoder (VAE) network.

[0147] Figure 4 is a schematic diagram of a deep neural network structure provided in an embodiment of this application. Referring to Figure 4, the deep neural network includes an encoding unit and a decoding unit. The encoding unit is used to perform the dimensionality reduction processing in this embodiment (described in detail below), and can also be called a dimensionality reduction unit. The decoding unit is used to perform the dimensionality increase processing in this embodiment, and can also be called a dimensionality increase unit. h in Figure 4... t f represents hidden state data. dkv (h t () indicates dimensionality reduction. This represents the dimensionality reduction result, such as the result of the first intermediate operation. This indicates dimensionality increase processing, such as first dimensionality increase processing or second dimensionality increase processing. and This represents the keys and values ​​obtained from the dimensionality reduction process. For example, the first key-value pair. The dimension of the hidden state data might be 4096, while the dimension of the dimensionality-reduced result might be 512, meaning the amount of data in the dimensionality-reduced result is much smaller than that of the hidden state data.

[0148] It should be understood that Figure 4 uses a three-layer neural network as an example, where both the dimensionality reduction and dimensionality increase units are three-layer neural networks. Figure 4 is not intended to limit the number of layers in the deep neural network in this embodiment; that is, this embodiment does not limit the number of layers in the deep neural network. More layers result in stronger learning ability and faster convergence speed; fewer layers result in higher processing efficiency. The number of layers in a deep neural network can be flexibly set according to actual conditions.

[0149] In some embodiments, the network model includes a first type of attention mechanism module and a second type of attention mechanism module. The first type of attention mechanism module performs dimensionality reduction processing on the hidden state data of the input information to obtain a dimensionality-reduced result, i.e., an intermediate computation result with a smaller data volume, and performs dimensionality increase on the dimensionality-reduced result to obtain key-value pairs with a larger data volume. The second type of attention mechanism module processes the hidden state data of the input information to obtain key-value pairs. That is, the first type of attention mechanism module has the function of reducing the amount of data in the key-value cache, while the second type of attention mechanism module does not have this function. The first type of attention mechanism module is the main module for achieving the effects of the embodiments of this application and can be called a new module (or a new model), while the second type of attention mechanism module can be called the original module or the old module (or the old model).

[0150] Therefore, the key-value cache may store intermediate computation results generated using both the first type of attention mechanism module and the second type of attention mechanism module. For example, if inference is performed using the second type of attention mechanism module within a first time period, all intermediate computation results obtained during this period are generated using the second type of attention mechanism module, and the key-value cache stores these intermediate computation results from the first time period. If inference is performed using the first type of attention mechanism module within a second time period following the first time period, all intermediate computation results obtained during this period are generated using the first type of attention mechanism module, and the key-value cache also stores these intermediate computation results from the second time period. The first time period may include the period before the first type of attention mechanism module has been fully trained, and the second time period may include the period after the first type of attention mechanism module has been fully trained.

[0151] When the key-value cache stores intermediate computation results generated by the first type of attention mechanism module and the second type of attention mechanism module, the intermediate computation results in the key-value cache also have indication information of the generation method. This indication information indicates whether the corresponding intermediate computation result was generated by the first type of attention mechanism module or the second type of attention mechanism module. For example, the aforementioned first intermediate computation result also has first indication information, which indicates whether the first intermediate computation result was generated by the first type of attention mechanism module or the second type of attention mechanism module. Before performing dimensionality upscaling on the first intermediate computation result through the network model to obtain the first key-value pair, the first indication information is obtained. If the first indication information indicates that the first intermediate computation result was generated by the first type of attention mechanism module, then the first intermediate computation result is subjected to dimensionality upscaling through the first type of attention mechanism module to obtain the first key-value pair.

[0152] In some implementations, the network model includes a first-type attention mechanism module. The indication information for the generation method of intermediate computation results can be an indicator label, which can be a first label value or a second label value. The first label value indicates that the corresponding intermediate computation result was generated using the first-type attention mechanism module, and the second label value indicates that the corresponding intermediate computation result was generated using the second-type attention mechanism. As an example, the first label value and the second label value are 1 and 0, respectively. Of course, besides the indicator label, other methods can also be used to represent the above-mentioned generation method indication information. This application does not limit this; for example, the indication information can be an identifier of the first-type attention mechanism module or the second-type attention mechanism module, that is, the identifier of the attention mechanism module used to generate the corresponding intermediate computation result.

[0153] In another implementation, the network model includes multiple attention mechanism modules corresponding to different application domains. Each of these modules corresponds one-to-one with the application domain. Each attention mechanism module has the function of reducing the amount of data in the key-value cache. Specifically, each module performs dimensionality reduction on the hidden state data of the input information in its corresponding domain to obtain intermediate computation results with less data, and performs dimensionality increase on these intermediate computation results to obtain key-value pairs. It should be understood that these multiple attention mechanism modules include the first type of attention mechanism module mentioned above; that is, the first type of attention mechanism module is one of these multiple attention mechanism modules. The network model also includes a second type of attention mechanism module. Based on this, the indication information for the generation method of intermediate computation results indicates which module among the multiple attention mechanism modules and the second type of attention mechanism module was used to generate the corresponding intermediate computation result.

[0154] As an example, the indication information includes a first type of indication label and a second type of indication label. The first type of indication label indicates whether the corresponding intermediate operation result was generated using the aforementioned multiple attention mechanism modules or using the second type of attention mechanism module. The second type of indication label indicates, if the corresponding intermediate operation result was generated using the aforementioned multiple attention mechanism modules, which specific attention mechanism module was used. As another example, the indication information could be an identifier of the attention mechanism module used to generate the corresponding intermediate operation result.

[0155] In this implementation, the first type of attention mechanism module used to process the first intermediate operation result is one of the multiple attention mechanism modules indicated by the first instruction information.

[0156] It should be understood that if the first intermediate operation result is generated using the second type of attention mechanism module, then the first intermediate operation result is a key-value pair, which can be processed by the second type of attention mechanism module. For specific implementation methods, please refer to relevant technologies.

[0157] In other embodiments, the network model includes a first type of attention mechanism module but not a second type of attention mechanism module, and the intermediate computation results in the key-value cache are all generated using the first type of attention mechanism module. Based on this, after the first input information hits the first intermediate computation result in the key-value cache, the first intermediate computation result is directly subjected to dimensionality-upgrading processing through the first type of attention mechanism module.

[0158] In these embodiments, intermediate computation results in the key-value cache do not need to have information indicating the generation method. Alternatively, considering that the first type of attention mechanism module may be updated (e.g., version upgrade), or multiple attention mechanism modules corresponding to different application domains may be added, or there may be subsequent compatibility (or fallback) to the second type of attention mechanism module, then the intermediate computation results in the key-value cache also need to have information indicating the generation method.

[0159] In some embodiments, the network model includes multiple attention mechanism modules corresponding to different application domains (including a first type of attention mechanism module), but excludes a second type of attention mechanism module. The intermediate computation results in the key-value cache are all dimensionality reduction results with relatively small data volumes, and these intermediate computation results also contain indication information about their generation method. Based on this, after the first input information hits the first intermediate computation result in the key-value cache, first indication information can be obtained, and the first type of attention mechanism module can be determined from the multiple attention mechanism modules based on the first indication information.

[0160] As described above, the network model in this application embodiment has at least four specific implementation methods. In the first specific implementation method, the network model includes a new module and an old module; in the second specific implementation method, the network model includes a new module but not an old module; in the third specific implementation method, the network model includes multiple new modules (corresponding to multiple application domains) but not an old module; in the fourth specific implementation method, the network model includes multiple new modules and an old module. In different specific implementation methods, the types of intermediate operation results in the key-value cache are different, and the metadata of the intermediate operation results also differs. The metadata includes indication information of the above-mentioned generation method, or, may also include the above-mentioned index.

[0161] This application does not limit the classification of application fields. As an example, the multiple application fields include the fields of daily life, medical care, and education.

[0162] Step 303: Based on the first key-value pair, determine the first output result through the network model. The first output result is the output result corresponding to the first input information.

[0163] As described above in the computational process of the attention mechanism, it involves not only key-value pairs but also a query matrix. Therefore, in addition to obtaining the first key-value pair through step 302, the network model also processes the first input information to obtain the first query matrix. The specific implementation can be found in relevant technologies and will not be elaborated upon here.

[0164] After obtaining the first query matrix and the first key-value pair, the network model can determine the first output result based on the first key-value pair and the first query matrix. For specific implementation details, please refer to relevant technologies. As an example, the first score matrix is ​​obtained by calculating the dot product of the transpose of the first query matrix and the first key using either the first or second type of attention mechanism module. After scaling the first score matrix, the softmax function is applied to obtain the first weight matrix. The first value is then multiplied by the first weight matrix to obtain the first output matrix. This first output matrix is ​​the output matrix of either the first or second type of attention mechanism module. Based on the first output matrix, the output processing module of the network model determines the first output result.

[0165] Steps 302 and 303 above describe the case where the first input information hits the key-value cache. Next, we will describe the case where the first input information does not hit the key-value cache.

[0166] Figure 5 is a flowchart of another method for determining the output result provided in an embodiment of this application. Referring to Figure 5, after obtaining the first input information, if the first input information does not hit the key-value cache, the following steps are performed.

[0167] Step 501: If the first input information does not hit the key-value cache, the hidden state data of the first input information is determined through the network model to obtain the first hidden state data.

[0168] The specific implementation method for determining the hidden state data can be found in relevant technologies, which will not be discussed in detail here.

[0169] Step 502: Based on the first hidden state data, determine the key-value pairs of the first input information through the network model.

[0170] In the case where the network model includes a first-class attention mechanism module (i.e., the new module is one and does not include the old module), referring to Figure 6, step 502 includes the following steps a1 to a2.

[0171] Step a1: Dimensionality reduction of the first hidden state data is performed using the first type of attention mechanism module to obtain the second intermediate computation result.

[0172] As mentioned above, in one implementation method, the dimensionality reduction result can be a single data point, while in another, the result can be two data points, with K and V each corresponding to a dimensionality reduction result. These two implementation methods will be introduced separately below.

[0173] In the first implementation, the first hidden state data is dimensionality reduced by the first type of attention mechanism module to obtain the second intermediate operation result, which is a data.

[0174] For example, the first type of attention mechanism module includes a dimensionality reduction unit, which performs dimensionality reduction processing on the first hidden state data.

[0175] In the second implementation, the first hidden state data undergoes a first dimensionality reduction process using a first type of attention mechanism module to obtain a first key dimensionality reduction result. Then, a second dimensionality reduction process is performed on the first hidden state data to obtain a first value dimensionality reduction result. The second intermediate computation result includes both the first key dimensionality reduction result and the first value dimensionality reduction result.

[0176] For example, the first type of attention mechanism module includes a first dimensionality reduction unit and a second dimensionality reduction unit. The first dimensionality reduction unit performs a first dimensionality reduction process on the first hidden state data, and the second dimensionality reduction unit performs a second dimensionality reduction process on the first hidden state data. The dimensionality reduction parameters of the first dimensionality reduction unit and the second dimensionality reduction unit are different.

[0177] It is worth noting that the first implementation method described above is computationally simple and time-efficient because it only requires one dimensionality reduction process on the first hidden state data. The second implementation method, however, can perform two dimensionality reduction processes on the first hidden state data. This allows for flexible setting of the dimensionality reduction parameters used in these two processes. Thus, different dimensionality reduction parameters (such as two sets of parameters) can be used to control whether the dimensions of the key-value dimensionality reduction results are the same or different. Furthermore, because the two-set dimensionality reduction parameter scheme is more flexible, allowing for dimensionality reduction based on the importance of K and V respectively, its performance is relatively better.

[0178] In one implementation, the dimensions of the first key-based dimensionality reduction result and the first value-based dimensionality reduction result are different. For example, considering that the importance of the key (such as accuracy and precision requirements) may be higher than the importance of the value, the dimension of the first key-based dimensionality reduction result is higher than the dimension of the first value-based dimensionality reduction result. Of course, it is also possible to set the dimensions of the key-based dimensionality reduction result and the value-based dimensionality reduction result to be the same, or the dimension of the key-based dimensionality reduction result to be lower than the dimension of the value-based dimensionality reduction result, depending on the requirements.

[0179] The dimensionality reduction process described above is performed by a single-layer neural network or a deep neural network. Here, a deep neural network refers to a multi-layer neural network. This application does not limit the specific structure of the single-layer neural network or the deep neural network, nor does it limit the number of layers in the deep neural network. As an example, the single-layer neural network can be a LoRA network, and the deep neural network model can be an AE network.

[0180] In other words, the first type of attention mechanism module can include a single-layer neural network or a deep neural network to perform the dimensionality reduction process described above.

[0181] As an example, in an implementation where the first type of attention mechanism module includes a dimensionality reduction unit, the dimensionality reduction unit may include a single-layer neural network or a deep neural network. In an implementation where the first type of attention mechanism module includes a first dimensionality reduction unit and a second dimensionality reduction unit, the first dimensionality reduction unit includes a first single-layer neural network or a first deep neural network, and the second dimensionality reduction unit includes a second single-layer neural network or a second deep neural network.

[0182] Step a2: The second intermediate operation result is upgraded using the first type of attention mechanism module to obtain the key-value pairs of the first input information.

[0183] For ease of distinction, the key-value pair of the first input information can also be called the second key-value pair.

[0184] In the implementation of the second intermediate operation result being a data, the second intermediate operation result is subjected to a first dimensionality increase process by the first type of attention mechanism module to obtain the key in the key-value pair of the first input information (i.e., the key in the second key-value pair), and the second intermediate operation result is subjected to a second dimensionality increase process by the first type of attention mechanism module to obtain the value in the key-value pair of the first input information (i.e., the value in the second key-value pair).

[0185] As an example, the first type of attention mechanism module includes a first dimensionality-upgrading unit and a second dimensionality-upgrading unit. The first dimensionality-upgrading unit performs a first dimensionality-upgrading process on the second intermediate computation result, and the second dimensionality-upgrading unit performs a second dimensionality-upgrading process on the second intermediate computation result. The dimensionality-upgrading parameters of the first and second dimensionality-upgrading units are generally different, although it is possible for them to be the same.

[0186] In the implementation method where the second intermediate operation result includes two data (i.e., the first key dimensionality reduction result and the first value dimensionality reduction result), the first key dimensionality reduction result is upgraded by the first type of attention mechanism module to obtain the key in the second key-value pair, and the first value dimensionality reduction result is upgraded by the first type of attention mechanism module to obtain the value in the second key-value pair.

[0187] As an example, the first type of attention mechanism module includes a first dimensionality-upgrading unit and a second dimensionality-upgrading unit. The first dimensionality-upgrading unit performs dimensionality-upgrading processing on the first key dimensionality reduction result, and the second dimensionality-upgrading unit performs dimensionality-upgrading processing on the first value dimensionality reduction result.

[0188] In the case where the network model includes a first type of attention mechanism module and a second attention mechanism module (i.e., including a new module and an old module), referring to Figure 7, step 502 includes the following steps b1 to b4.

[0189] Step b1: Obtain the second instruction information, which indicates whether to use the first type of attention mechanism module or the second type of attention mechanism module for reasoning.

[0190] The second instruction information can be an instruction label or other forms of information, and this application embodiment does not limit this.

[0191] During the first time period, the second instruction information can instruct the use of a second type of attention mechanism module for inference; during the second time period, the second instruction information can instruct the use of a first type of attention mechanism module for inference. The first time period may include the period during which the first type of attention mechanism module has not yet been fully trained, and the second time period may include the period during which the first type of attention mechanism module has been fully trained.

[0192] If the second instruction indicates that the first type of attention mechanism module is used for reasoning, steps b2 and b3 are executed; if the second instruction indicates that the second type of attention mechanism module is used for reasoning, step b4 is executed.

[0193] Step b2: Dimensionality reduction of the first hidden state data is performed using the first type of attention mechanism module to obtain the second intermediate computation result.

[0194] Step b3: The second intermediate operation result is upgraded using the first type of attention mechanism module to obtain the key-value pairs of the first input information.

[0195] Step b4: Process the first hidden state data through the second type of attention mechanism module to obtain the key-value pairs of the first input information.

[0196] The specific implementation methods for steps b2 and b3 can be referred to the specific implementation methods for steps a1 and a2 above, and will not be repeated here. The specific implementation method for step b4 can be referred to relevant technologies, and will not be introduced in detail here.

[0197] In the case where the network model includes multiple attention mechanism modules (i.e., includes multiple new modules but not old modules), referring to Figure 8, step 502 includes steps c1 to c4 as follows.

[0198] Step c1: Determine the application domain to which the first input information belongs, and obtain the first application domain.

[0199] As an example, the network model also includes a classification module, which classifies the first input information to determine the application domain to which the first input information belongs.

[0200] Step c2: Determine the attention mechanism module corresponding to the first application domain among the multiple attention mechanism modules as the first type of attention mechanism module.

[0201] One attention mechanism module corresponds to one application domain, and different attention mechanism modules correspond to different application domains.

[0202] Step c3: Dimensionality reduction of the first hidden state data is performed using the first type of attention mechanism module to obtain the second intermediate operation result.

[0203] Step c4: The second intermediate operation result is upgraded using the first type of attention mechanism module to obtain the key-value pairs of the first input information.

[0204] The specific implementation methods of steps c3 and c4 can be referred to the specific implementation methods of steps a1 and a2 above, and will not be repeated here.

[0205] In the case where the network model includes multiple attention mechanism modules and a second type of attention mechanism module, referring to Figure 9, step 502 includes the following steps d1 to d6.

[0206] Step d1: Obtain the third instruction information, which indicates whether to use one of the multiple attention mechanism modules for reasoning or to use the second type of attention mechanism module for reasoning.

[0207] The third instruction information can be an instruction label or other forms of information, and this application embodiment does not limit this.

[0208] During the first time period, the third instruction information can instruct the use of a second type of attention mechanism module for inference; during the second time period, the third instruction information can instruct the use of one of multiple attention mechanism modules for inference. The first time period may include the time period during which the multiple attention mechanism modules have not yet been fully trained, and the second time period may include the time period during which the multiple attention mechanism modules have been fully trained.

[0209] If the third instruction indicates that the multiple attention mechanism modules are used for reasoning, steps d2 to d5 are executed; if the third instruction indicates that the second type of attention mechanism module is used for reasoning, step d6 is executed.

[0210] Step d2: Determine the application domain to which the first input information belongs, and obtain the first application domain.

[0211] Step d3: Determine the attention mechanism module corresponding to the first application domain among the multiple attention mechanism modules as the first type of attention mechanism module.

[0212] Step d4: Dimensionality reduction of the first hidden state data is performed using the first type of attention mechanism module to obtain the second intermediate operation result.

[0213] Step d5: The second intermediate operation result is upgraded using the first type of attention mechanism module to obtain the key-value pairs of the first input information.

[0214] Step d6: Process the first hidden state data through the second type of attention mechanism module to obtain the key-value pairs of the first input information.

[0215] The specific implementation methods of steps d2 to d5 can be referred to in the above description of steps a1 and a2, as well as steps c1 to c4. The specific implementation method of step d6 can be referred to in the above description of step b4. It will not be repeated here.

[0216] In the embodiments shown in Figures 8 and 9 above, since the network model includes multiple attention mechanism modules, the current inference task can be scheduled and distributed based on the application domain to which the current input information belongs, thereby improving inference efficiency. For example, these multiple attention mechanism modules (and the second type of attention mechanism module) can be deployed on multiple computer devices, and the load can be distributed among inference tasks in different application domains through scheduling and distribution.

[0217] It should be understood that, ideally, the key-value pairs of the first input information obtained using the first type of attention mechanism module (denoted as the second key-value pair) are the same as the key-value pairs of the first input information obtained using the second type of attention mechanism module (denoted as the third key-value pair). However, in practical applications, the second key-value pairs and the third key-value pairs may have some differences. The magnitude of this difference is related to the training accuracy or performance of the two types of attention mechanism modules. For example, the higher the training accuracy of the first type of attention mechanism module, the smaller the difference.

[0218] In cases where the second intermediate computation result is generated using the first type of attention mechanism module, in some embodiments (e.g., the network model also includes a second type of attention mechanism module), a fourth indication is stored. This fourth indication indicates that the second intermediate computation result was generated using the first type of attention mechanism module. As an example, the fourth indication can be stored in the metadata of the second intermediate computation result.

[0219] Furthermore, after reducing the dimensionality of the first hidden state data using the first type of attention mechanism module to obtain the second intermediate operation result, the second intermediate operation result is stored in a key-value cache, and an index of the second intermediate operation result is constructed. The index of the second intermediate result can be constructed based on the first input information; the index construction method can refer to relevant technologies, which will not be elaborated upon here.

[0220] After processing the first hidden state data through the second type of attention mechanism module to obtain the key-value pairs of the first input information, these key-value pairs are stored in the key-value cache. That is, the higher-dimensional key-value pairs are directly stored. Furthermore, it is also necessary to store the index of the fifth indication information and the key-value pairs of the first input information, for example, by storing the fifth indication information and the index in the metadata of the key-value pairs of the first input information. The fifth indication information indicates that the third intermediate operation result is generated using the second type of attention mechanism module, and the third intermediate operation result is the key-value pair of the first input information obtained through the second type of attention mechanism module.

[0221] Alternatively, after processing the first hidden state data using the second type of attention mechanism module to obtain the key-value pairs of the first input information, the first type of attention mechanism module is also used to perform dimensionality reduction on the first hidden state data to obtain the second intermediate operation result, which is then stored in the key-value cache. That is, inference is performed using the second type of attention mechanism module, while the first type of attention mechanism module is used to obtain and store the smaller amount of intermediate operation result. Furthermore, the indexes of the key-value pairs of the fourth indication information and the first input information also need to be stored.

[0222] In some embodiments, the metadata of intermediate operation results in the key-value cache includes not only index and generation method indication information, but also generation time, version information of the attention mechanism module used to generate the corresponding intermediate operation results, etc. This application embodiment does not limit this.

[0223] In the above embodiments, there is only one second-type attention mechanism module, meaning the network model includes a legacy module that exhibits good generalization across various application domains. This means the second-type attention mechanism module can process the hidden state data of input information in any application domain. In other embodiments, the network model may include multiple legacy modules, such as multiple second-type attention mechanism modules. Different second-type attention mechanism modules correspond to different application domains, and each module processes the hidden state data of input information in its corresponding application domain. Therefore, before using a second-type attention mechanism module for inference, it is necessary to first determine the application domain to which the current input information belongs, thereby determining which second-type attention mechanism module to use. The indication information in the above-mentioned generation method should also indicate which second-type attention mechanism module was used to generate the corresponding intermediate computation result.

[0224] Step 503: Based on the key-value pairs of the first input information, determine the first output result through the network model.

[0225] The specific implementation of step 503 is similar to that of step 303, and will not be described in detail here.

[0226] Next, please refer to Figure 10 to provide an exemplary explanation of the above embodiments.

[0227] Figure 10 is a schematic diagram of some modules in a network model provided in an embodiment of this application. Referring to Figure 10, the network model includes a multi-layer network structure, and each layer includes a first type of attention mechanism module, a second type of attention mechanism module, and an MLP. The output data of the first type of attention mechanism module or the second type of attention mechanism module of the current layer is the input data of the MLP of the current layer, and the output data of the MLP of the current layer is the input data of the next layer.

[0228] First, we will introduce the case where inference is performed using the second type of attention mechanism module. The input data for the second type of attention mechanism module is the hidden state data (denoted as h). t The second type of attention mechanism module uses W V W K and W Q The three processing units represented respectively handle h t After processing, the value is obtained (denoted as v). t,i ), key (denoted as k) t,i ) and query matrix (denoted as q) t,i According to the formula Calculate the weight matrix (denoted as P). t Then calculate P. t and v t,i The product of these components yields the output matrix (denoted as h) of the second type of attention mechanism module. mlp,t ), h mlp,t This is the input data for the MLP.

[0229] Next, we will introduce the case where inference is performed using the first type of attention mechanism module. The input data for the first type of attention mechanism module is h. t The first type of attention mechanism module uses W DKV The dimensionality reduction unit represented by h t The intermediate calculation results are obtained through processing (denoted as...). This can also be called the hidden space (the intermediate operation result can be stored in the key-value cache), and then processed through W. UK The first up-dimensional unit pair represented Processing yields the key (denoted as...). ), and through W UV The second up-dimensional unit pair is characterized Process it to get the value (denoted as ). In addition, it is also through W Q The processing units represented are respectively for h t After processing, the query matrix is ​​obtained (denoted as q). t,i Alternatively, it can be obtained through the W module of the second type of attention mechanism. Q The processing unit represented by q t,i Then, according to the formula Calculate the weight matrix (denoted as ) ), then calculate and The product of these two matrices yields the output matrix of the first type of attention mechanism module (denoted as ). ), This is the input data for the MLP.

[0230] As described above, the main difference in the inference process between the first and second type of attention mechanism modules lies in their approaches. The first type of attention mechanism module first performs dimensionality reduction on the hidden state data, and then performs dimensionality upscaling to obtain key-value pairs. The key-value cache can store the data obtained after dimensionality reduction, thus reducing storage space usage. In contrast, the second type of attention mechanism module directly processes the hidden state data to obtain key-value pairs, and can only store key-value pairs with a large amount of data in the key-value cache, resulting in a larger storage space usage.

[0231] It should be understood that the input data of the MLP is used to input into the MLP, and the MLP processes h. mlp,t or After processing, the data is input into the next layer of the network structure. For specific implementation details, please refer to relevant technologies. If the next layer's network structure is the same as the current layer's network structure, then the next layer's processing method for the MLP's output data is similar to the current layer's processing method for the hidden state data; therefore, it will not be repeated here.

[0232] W in Figure 10 DKV This can represent f in Figure 4 dkv The parameter of (), W UK and W UV f in Figure 4 can be represented ukv The parameters of (). In Figure 10, Therefore, it can be seen that the data compression rate of the key-value cache in this embodiment can reach dimension. / 2dimension(k t,i ), where dimension This represents the dimension of the intermediate computation result obtained by dimensionality reduction processing (such as using the dimensionality reduction unit in the first type of attention mechanism module), 2dimension(k) t,i ) represents the dimension of the intermediate computation result obtained without dimensionality reduction processing (such as using a second type of attention mechanism module).

[0233] The above describes the implementation process of inference using the first type of attention mechanism module. Before using the first type of attention mechanism module for inference, it is necessary to train the first type of attention mechanism module, which will be introduced next.

[0234] In one implementation, before inference using the first type of attention mechanism module—for example, before dimensionality reduction of the first hidden state data using the first type of attention mechanism module—the first type of attention mechanism module is trained online using a second type of attention mechanism module. That is, during the inference process of the second type of attention mechanism module, the first type of attention mechanism module is trained online using data from its inference process.

[0235] Taking the determination of the output result corresponding to the second input information (called the second output result) through the second type of attention mechanism module as an example, after obtaining the second input information of the network model, the hidden state data of the second input information is determined to obtain the second hidden state data. The second input information is the input information before the first input information. The second hidden state data is processed by the second type of attention mechanism module to obtain the key-value pair of the second input information (denoted as the fourth key-value pair). Based on the key-value pair of the second input information, the second output result is determined by the second type of attention mechanism module, and the first type of attention mechanism module is trained using the second hidden state data.

[0236] It should be understood that, in the process of determining the second output result, the second hidden state data is also processed through a second type of attention mechanism module to obtain the query matrix corresponding to the second input information (denoted as the second query matrix). Based on the key-value pairs of the second input information and the corresponding query matrix, the second output result is determined through the second type of attention mechanism module. For specific implementation details, please refer to the embodiment in Figure 10 or related technologies; these details will not be elaborated here.

[0237] In this embodiment of the application, training a first type of attention mechanism module using second hidden state data includes: performing dimensionality reduction processing on the second hidden state data through the first type of attention mechanism module to obtain sample dimensionality reduction results; performing dimensionality increase processing on the sample dimensionality reduction results through the first type of attention mechanism module to obtain reconstructed key-value pairs; determining a loss value based on the reconstructed key-value pairs; and updating the parameters of the first attention mechanism module based on the loss value.

[0238] One implementation of dimensionality reduction processing on the second hidden state data to obtain the sample dimensionality reduction result is as follows: perform dimensionality reduction processing on the second hidden state data once to obtain a single data point, which is the sample dimensionality reduction result. Correspondingly, one implementation of dimensionality increase processing on the sample dimensionality reduction result to obtain the reconstructed key-value pairs is as follows: perform a first dimensionality increase processing on the sample dimensionality reduction result to obtain the key (denoted as the reconstructed key) in the reconstructed key-value pair; perform a second dimensionality increase processing on the sample dimensionality reduction result to obtain the value (denoted as the reconstructed value) in the reconstructed key-value pair.

[0239] For example, the first type of attention mechanism module includes a dimensionality reduction unit, which performs dimensionality reduction processing on the second hidden state data to obtain the sample dimensionality reduction result. The first type of attention mechanism module also includes a first dimensionality increase unit and a second dimensionality increase unit. The first dimensionality increase unit performs dimensionality increase processing on the sample dimensionality reduction result to obtain the reconstructed key, and the second dimensionality increase unit performs dimensionality increase processing on the sample dimensionality reduction result to obtain the reconstructed value.

[0240] Another way to reduce the dimensionality of the second hidden state data to obtain the sample dimensionality reduction result is as follows: perform a first dimensionality reduction on the second hidden state data to obtain the key sample dimensionality reduction result, and then perform a second dimensionality reduction on the second hidden state data to obtain the value sample dimensionality reduction result. That is, the sample dimensionality reduction result includes both the key sample dimensionality reduction result and the value sample dimensionality reduction result. Correspondingly, one way to increase the dimensionality of the sample dimensionality reduction result to obtain the reconstructed key-value pairs is as follows: increase the dimensionality of the key sample dimensionality reduction result to obtain the reconstructed key, and increase the dimensionality of the value sample dimensionality reduction result to obtain the reconstructed value.

[0241] For example, the first type of attention mechanism module includes a first dimensionality reduction unit and a second dimensionality reduction unit. The first dimensionality reduction unit performs a first dimensionality reduction process on the second hidden state data to obtain the dimensionality reduction result of the key samples, and the second dimensionality reduction unit performs a second dimensionality reduction process on the second hidden state data to obtain the dimensionality reduction result of the value samples. The first type of attention mechanism module also includes a first dimensionality increase unit and a second dimensionality increase unit. The first dimensionality increase unit performs a dimensionality increase process on the dimensionality reduction result of the key samples to obtain the reconstructed keys, and the second dimensionality increase unit performs a dimensionality increase process on the dimensionality reduction result of the value samples to obtain the reconstructed values.

[0242] There are multiple ways to determine the loss value based on the reconstruction of key-value pairs, and these multiple methods will be introduced below.

[0243] One approach to determining the loss value is to base it on the fourth key-value pair and the reconstructed key-value pair. The fourth key-value pair is determined based on the second hidden state data; for example, it can be obtained by processing the second hidden state data using a second-type attention mechanism module.

[0244] As an example, still using Figure 10, we can follow the formula... To determine the loss value (i.e., loss). Here, L() represents the loss function, and the specific form and parameters of the loss function are not limited in the embodiments of this application.

[0245] A second approach to determining the loss value is as follows: Based on the reconstructed key and the second query matrix, a reconstructed weight matrix is ​​determined. The loss value is then determined based on the second weight matrix, the reconstructed weight matrix, the values ​​in the fourth key-value pair, and the reconstructed value. The second query matrix is ​​determined based on the second hidden state data; for example, it can be obtained by processing the second hidden state data using a second-type attention mechanism module. Similarly, the second weight matrix is ​​determined based on the keys in the fourth key-value pair and the second query matrix; for instance, it can be obtained by processing the keys in the fourth key-value pair and the second query matrix using a second-type attention mechanism module.

[0246] As an example, still using Figure 10, we can follow the formula... To determine the loss value (i.e., loss). Here, L() represents the loss function, and the specific form and parameters of the loss function are not limited in the embodiments of this application.

[0247] The third method for determining the loss value is as follows: Based on the reconstruction key and the second query matrix, determine the reconstruction weight matrix; based on the reconstruction weight matrix and the reconstructed values, determine the reconstruction output matrix; and based on the second output matrix and the reconstruction output matrix, determine the loss value. Specifically, the second query matrix is ​​determined based on the second hidden state data; the second output matrix is ​​determined based on the second weight matrix and the values ​​in the fourth key-value pair; and the second weight matrix is ​​determined based on the keys in the second key-value pair and the second query matrix. The methods for determining the second query matrix, the second output matrix, and the second weight matrix can be referred to the relevant content above, and will not be repeated here.

[0248] As an example, still using Figure 10, we can follow the formula... To determine the loss value (i.e., loss). Here, L() represents the loss function, and the specific form and parameters of the loss function are not limited in the embodiments of this application.

[0249] The convergence speed and training accuracy of the three methods for determining the loss value described above may differ. For example, the first method has a slower convergence speed but higher training accuracy, while the third method has a faster convergence speed but slightly lower training accuracy. The method for determining the loss value can be flexibly set according to actual needs. Besides the three methods mentioned above, the loss value can also be determined using other data or methods; this application does not limit this approach.

[0250] As described above, the first type of attention mechanism module in this embodiment can be used to perform dimensionality reduction processing on the hidden state data of the input information in the first application domain. That is, the first type of attention mechanism module is the attention mechanism module corresponding to the first application domain. Based on this, during the online training of the first type of attention mechanism module using the second type of attention mechanism module, it is first necessary to determine whether the input information (e.g., the second input information) of the second type of attention mechanism module belongs to the first application domain. If the input information of the second type of attention mechanism module belongs to the first application domain, the first type of attention mechanism module is trained online using the second type of attention mechanism module. For example, if the second input information belongs to the first application domain, the first type of attention mechanism module is trained online using the second hidden state data.

[0251] If the input information for the second type of attention mechanism module does not belong to the first application domain, the first type of attention mechanism module will not be trained temporarily. For example, if the second input information does not belong to the first application domain, it is not conducive to training the first type of attention mechanism module with the second hidden state data; that is, the training process skips the second hidden state data.

[0252] When the input information of the second type of attention mechanism module changes to belong to the first application domain, the hidden state data of the current input information is used to continue to train the first type of attention mechanism module online.

[0253] Of course, in some other embodiments, the first type of attention mechanism module may not correspond to any specific application domain. That is, the first type of attention mechanism module can be trained using the hidden state data of any input information from the second type of attention mechanism module. In this way, the generalization ability of the trained first type of attention mechanism module is better.

[0254] If the loss value is less than the loss threshold, or the number of training iterations reaches the threshold, the training of the first type of attention mechanism module is considered complete. After the first type of attention mechanism module is trained, it can be deployed and applied online. For example, the first type of attention mechanism module can replace the second type of attention mechanism module, or the dimensionality reduction and dimensionality expansion units in the first type of attention mechanism module can replace the first processing unit (e.g., W in Figure 10) in the second type of attention mechanism module. K The represented unit) and the second processing unit (e.g., W in Figure 10) V (The units represented). Among them, the first processing unit is used to process the hidden state data to obtain the key, and the second processing unit is used to process the hidden state data to obtain the value.

[0255] As discussed above, the network model can include attention mechanism modules corresponding to multiple application domains. Any one of these attention mechanism modules can be trained using the methods described above. For example, the second type of attention mechanism module can be used for online training of any one of these attention mechanism modules. This addresses the problem of insufficient inference accuracy in scenarios where the application domains of the training data are unevenly distributed. Specifically, by training multiple attention mechanism modules, inference accuracy and precision across various application domains can be improved, while also achieving a balance in generalization.

[0256] In one possible implementation, after obtaining the current input information of the network model, the application domain to which the current input information belongs is determined, thus obtaining the target application domain. During the reasoning process using the second type of attention mechanism module on the current input information, the hidden state data of the current input information is used to train the attention mechanism module corresponding to the target application domain online. Here, the current input information refers to the input information within the online training period, which is the period during which the multiple attention mechanism modules have not yet completed training.

[0257] It should be understood that the current input information can be the second input information mentioned above, or it can be other input information.

[0258] Furthermore, the training completion times of the aforementioned multiple attention mechanism modules can be different or the same, and this application embodiment does not impose any restrictions on this. Taking the training termination condition of the aforementioned multiple attention mechanism modules all including the corresponding loss value being less than the loss threshold (the loss thresholds corresponding to the multiple attention mechanism modules can be the same or different) as an example, the training completion times of these multiple attention mechanism modules are generally different. If none of the multiple attention mechanism modules have been trained, the application domain to which the current input information belongs can be the application domain corresponding to any of the attention mechanism modules that have not been trained. In this case, the hidden state data of the current input information can be used to train the attention mechanism module of the target application domain online. When there is a trained attention mechanism module among the multiple attention mechanism modules, the application domain to which the current input information belongs may be the application domain corresponding to an untrained attention mechanism module or the application domain corresponding to a trained attention mechanism module. If the application domain to which the current input information belongs is the application domain corresponding to an untrained attention mechanism module, then the attention mechanism module of the target application domain can be trained online using the hidden state data of the current input information. If the application domain to which the current input information belongs is the application domain corresponding to a trained attention mechanism module, then it is not conducive to training any attention mechanism module using the hidden state data of the current input information, that is, the training process skips the hidden state data of the current input information.

[0259] As can be seen from the above, the network model can include a multi-layer network structure, and each layer of the network structure can have an attention mechanism module. Based on this, a bypass first-type attention mechanism module can be added to the second-type attention mechanism module in some or all layers, thereby reducing the storage space occupied by the intermediate computation results of some or all layers.

[0260] Figure 11 is a training diagram of a first type of attention mechanism module provided in an embodiment of this application. Referring to Figure 11, taking a network model including 32 layers of Llama3 8B as an example, the 8th layer in the middle is selected for training. During the entire training process, the parameters of the main model (i.e., including the second type of attention mechanism module) are frozen (i.e., unchanged) and it is only responsible for inference. While the main model is performing inference, the new model (such as the first type of attention mechanism module) is in a learning state (i.e., a training state) and does not participate in the inference of the main model.

[0261] In this model, the output data of layer 7 is the input data of layer 8, which is h. t t is the token vector at the current time. h t The input data is fed into the Attention module (i.e., the second type of attention mechanism module) of the main model and simultaneously fed into the new model as learning input data. The new model uses this input data to calculate the corresponding output through a neural network, compares it with the output of the main model to calculate the loss, and performs backpropagation to update the parameters of the new model.

[0262] Figure 12 is a schematic diagram of inference for a first type of attention mechanism module provided in an embodiment of this application. Taking the network model shown in Figure 11 as an example, referring to Figure 12, after the new model is trained, inference is performed using the new model. During inference, the Attention module of layer 8 of the main model is disabled, i.e., it is in a useless state. t The data is only input into the new model, and inference is performed through the new model to obtain intermediate computational results with lower dimensionality, as well as the output data of the new model (e.g., ...). The output data of the new model is returned to the main model's MLP to continue inference.

[0263] The inference requests in Figures 11 and 12 carry input information, such as questions entered by the user. Word embeddings are used to convert the input information (such as text, images, videos, etc.) into vectors so that the network model can process them.

[0264] The online training methods described in the above embodiments do not require a dedicated training dataset, lowering the training threshold and improving inference accuracy. In other embodiments, the first type of attention mechanism module (or multiple attention mechanism modules) can be trained offline, for example, using an offline sample set. This offline sample set can be domain-independent, including sample data from multiple application domains, thus obtaining a first type of attention mechanism module with good generalization. Alternatively, the offline dataset can be domain-specific; for example, the offline dataset used to train the first type of attention mechanism module may include sample data from the first application domain but exclude data from other application domains, resulting in a first type of attention mechanism module specific to its application domain, making the inference application more targeted. In yet another embodiment, the first type of attention mechanism module (or multiple attention mechanism modules) can be trained offline first, and then the offline-trained first type of attention mechanism module can be trained online, thereby improving the inference performance of the first type of attention mechanism module.

[0265] The embodiments of this application do not limit the training device for the first type of attention mechanism module. The training device may be a computer device that uses the second type of attention mechanism module for inference, or other devices.

[0266] As described above, the first type of attention mechanism module in this application embodiment is compatible with the second type of attention mechanism module. Based on this, if, for some reason, the first type of attention mechanism module is no longer used or temporarily discontinued for inference (e.g., discontinuing the use of a certain version of the first type of attention mechanism module), it can revert to the second type of attention mechanism module. Before reverting, the first type of attention mechanism module can be used to perform dimensionality-upgrading on the low-dimensional intermediate computation results in the key-value cache to recover the original key-value pairs. This facilitates the reuse of the recovered original key-value pairs during subsequent inference using the second type of attention mechanism module. In other words, this application embodiment can achieve forward and backward compatibility, supporting the continuous and smooth evolution of new models. The compatibility and dynamic switching of the above attention mechanism modules can be supported by adding generation method indication information to the metadata of the intermediate computation results.

[0267] The technical solution of this application, for a main model that has already been trained, such as MHA or gene ontology annotation (GOA) models, can learn a low-dimensional key-value latent space by deploying a new model in a side path. This reduces the storage space occupied by the key-value cache, reduces the memory space occupied by data loading in the key-value cache, and reduces the loading latency of the key-value cache, thereby accelerating inference. This technical solution does not require retraining the entire network model; the training process of the new model does not affect the network model's performance of inference tasks, resulting in stronger usability and generalization.

[0268] Extensive experiments have shown that the key-value cache in the technical solution of this application can reduce the storage space occupied by more than 75% compared with related technologies, which significantly reduces the storage space occupied and helps to improve the overall operating performance of computer equipment.

[0269] In summary, the intermediate computation results in this embodiment can be dimensionality reduction results with a smaller data volume, which reduces the storage space occupied by the key-value cache. During inference, if the intermediate computation results are to be reused, their smaller data volume allows for faster loading, accelerating inference and improving the overall performance of the device. This is achieved by performing dimensionality upscaling on the dimensionality reduction results to obtain key-value pairs, which can then be reused to improve inference efficiency.

[0270] Figure 13 is a schematic diagram of a device structure for determining model output results provided in an embodiment of this application. This device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be the computer device shown in Figure 2. Referring to Figure 13, the device includes: a first acquisition module 1301, a dimensionality upgrade module 1302, and a first determination module 1303.

[0271] The first acquisition module 1301 is used to acquire the first input information of the network model;

[0272] The dimension-upgrading module 1302 is used to perform dimension-upgrading processing on the first intermediate calculation result through the network model if the first input information hits the first intermediate calculation result in the key-value cache, so as to obtain the first key-value pair. The data volume of the first intermediate calculation result is less than the data volume of the first key-value pair.

[0273] The first determining module 1303 is used to determine the first output result based on the first key-value pair through a network model. The first output result is the output result corresponding to the first input information.

[0274] In one possible implementation, the network model includes a first type of attention mechanism module and a second type of attention mechanism module; the device also includes:

[0275] The second acquisition module is used to acquire first indication information before performing dimensionality-upgrading processing on the first intermediate operation result through the network model to obtain the first key-value pair. The first indication information indicates whether the first intermediate operation result was generated by the first type of attention mechanism module or the second type of attention mechanism module.

[0276] The 1302 upgrade module is specifically used for:

[0277] If the first indication information indicates that the first intermediate operation result is generated using the first type of attention mechanism module, then the first intermediate operation result is subjected to dimensionality-upgrading processing through the first type of attention mechanism module to obtain the first key-value pair.

[0278] In one possible implementation, the device further includes:

[0279] The second determining module is used to determine the hidden state data of the first input information through the network model if the first input information does not hit the key value cache, and obtain the first hidden state data.

[0280] The third determination module is used to determine the second key-value pair based on the first hidden state data and through the network model.

[0281] The fourth determination module is used to determine the first output result based on the second key-value pair through the network model.

[0282] In one possible implementation, the network model includes a first type of attention mechanism module and a second type of attention mechanism module; the third determining module includes:

[0283] The first processing submodule is used to perform dimensionality reduction processing on the first hidden state data through a first type of attention mechanism module to obtain a second intermediate operation result, and to perform dimensionality increase processing on the second intermediate operation result through the first type of attention mechanism module to obtain key-value pairs of the first input information; or,

[0284] The second processing submodule is used to process the first hidden state data through the second type of attention mechanism module to obtain the key-value pairs of the first input information.

[0285] In one possible implementation, the device further includes:

[0286] The first storage module is used to store the second intermediate operation result in the key-value cache after the first hidden state data is reduced in dimensionality by the first type of attention mechanism module to obtain the second intermediate operation result.

[0287] In one possible implementation, the device further includes:

[0288] The third acquisition module is used to acquire second indication information before determining the second key-value pair based on the first hidden state data through the network model. The second indication information indicates whether to use the first type of attention mechanism module or the second type of attention mechanism module for inference.

[0289] In one possible implementation, the network model includes multiple attention mechanism modules, each attention mechanism module corresponding to an application domain, and the multiple attention mechanism modules include a first type of attention mechanism module;

[0290] The device also includes:

[0291] The fifth determining module is used to determine the application domain to which the first input information belongs before performing dimensionality reduction processing on the first hidden state data through the first type of attention mechanism module, and thus obtain the first application domain;

[0292] The sixth determining module is used to determine the attention mechanism module corresponding to the first application domain among the multiple attention mechanism modules as the first type of attention mechanism module.

[0293] In one possible implementation, the device further includes:

[0294] The online training module is used to train the first type of attention mechanism module online using the second type of attention mechanism module before performing dimensionality reduction processing on the first type of attention mechanism module for the first hidden state data.

[0295] In one possible implementation, the first type of attention mechanism module is the attention mechanism module corresponding to the first application domain;

[0296] The online training module includes:

[0297] The online training submodule is used to train the first type of attention mechanism module online when the input information of the second type of attention mechanism module belongs to the first application domain.

[0298] In one possible implementation, the second intermediate operation result includes the first key dimensionality reduction result and the first value dimensionality reduction result;

[0299] The first processing submodule is specifically used for:

[0300] Perform a first dimensionality reduction process on the first hidden state data to obtain the first key dimensionality reduction result;

[0301] The first hidden state data is subjected to a second dimensionality reduction process to obtain the first value dimensionality reduction result.

[0302] In one possible implementation, the dimensions of the first-key dimensionality reduction result and the first-value dimensionality reduction result are different.

[0303] In one possible implementation, the first intermediate operation result includes the second key dimensionality reduction result and the second value dimensionality reduction result;

[0304] Upgrade module 1302 includes:

[0305] The first dimension-upgrading submodule is used to perform dimension-upgrading processing on the second key dimension reduction result to obtain the key in the first key-value pair;

[0306] The second dimension-upgrading submodule is used to perform dimension-upgrading processing on the second value's dimension reduction result to obtain the value in the first key-value pair.

[0307] In another possible implementation, the dimensionality-upgrading module 1302 includes:

[0308] The third dimension-upgrading submodule is used to perform the first dimension-upgrading process on the first intermediate operation result to obtain the key in the first key-value pair;

[0309] The fourth dimension-upgrading submodule is used to perform a second dimension-upgrading process on the first intermediate operation result to obtain the value in the first key-value pair.

[0310] In one possible implementation, the above-mentioned dimensionality-up process is performed by a single-layer neural network or a deep neural network.

[0311] In this embodiment, the intermediate computation result can be a dimensionality reduction result with a small data volume, which reduces the storage space occupied by the key-value cache. During the inference process, if the intermediate computation result is to be reused, its loading speed is faster due to its small data volume, which can accelerate inference. By performing dimensionality upscaling on the dimensionality reduction result, key-value pairs can be obtained, thereby reusing the key-value pairs, improving inference efficiency, and enhancing the overall operating performance of the device.

[0312] It should be noted that the apparatus for determining model output results provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for determining model output results and the method for determining model output results provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0313] Figure 14 is a schematic diagram of a task execution device provided in an embodiment of this application. This device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be the one shown in Figure 2. In this embodiment, the network model includes a first type of attention mechanism module and a second type of attention mechanism module. Referring to Figure 14, the device includes: an inference module 1401 and an online training module 1402.

[0314] The reasoning module 1401 is used to perform reasoning tasks using the second type of attention mechanism module to obtain reasoning results; the specific implementation method can be referred to the relevant content in the above method embodiment, and will not be repeated here.

[0315] The online training module 1402 is used to perform online training of the first type of attention mechanism module using the second type of attention mechanism module. The trained first type of attention mechanism module is used to perform dimensionality reduction processing on the hidden state data of the input information of the network model to obtain intermediate operation results, and to perform dimensionality increase processing on the intermediate operation results to obtain key-value pairs of the input information. The intermediate operation results are used to store in the key-value cache for reuse. For specific implementation methods, please refer to the relevant content in the above method embodiment, which will not be repeated here.

[0316] In some implementations, the inference module 1401 is also used to perform an inference task using the first type of attention mechanism module after the first type of attention mechanism module has been trained, so as to obtain the inference result. Specific implementation details can be found in the relevant content of the method embodiments above, and will not be repeated here.

[0317] This application embodiment, for a main model that has already been trained (including the second type of attention mechanism module), can learn a low-dimensional key-value latent space by deploying a new model (including the first type of attention mechanism module) in the side path. This reduces the storage space occupied by the key-value cache and the memory usage of data loading in the key-value cache during subsequent inference using the first type of attention mechanism module, thus reducing the loading latency of the key-value cache and accelerating inference, thereby improving the overall performance of the device. This technical solution does not require retraining the entire network model; the training process of the new model does not affect the network model's inference task execution, resulting in stronger usability and generalization.

[0318] It should be noted that the task execution device provided in the above embodiments is only illustrated by the division of the above functional modules when performing tasks. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, some implementation methods of the task execution device provided in the above embodiments and the method embodiments for determining the model output results belong to the same concept. For details of the specific implementation process, please refer to the method embodiments, which will not be repeated here.

[0319] This application also provides a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the steps of the method for determining the output result of the model as shown in the above-described method embodiments.

[0320] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of the method for determining the output result of the model as shown in the above method embodiments.

[0321] This application also provides a computer program that, when run on a computer, causes the computer to perform the steps of the method for determining the output result of the model as shown in the above-described method embodiments.

[0322] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0323] It should be understood that "at least one" as mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., are not necessarily different.

[0324] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the input information involved in the embodiments of this application was obtained under full authorization.

[0325] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining the output result of a model, characterized in that, The method includes: Obtain the first input information of the network model; If the first input information hits the first intermediate operation result in the key-value cache, then the first intermediate operation result is upgraded through the network model to obtain the first key-value pair. The data volume of the first intermediate operation result is less than the data volume of the first key-value pair. Based on the first key-value pair, a first output result is determined through the network model. The first output result is the output result corresponding to the first input information.

2. The method as described in claim 1, characterized in that, The network model includes a first type of attention mechanism module and a second type of attention mechanism module; before performing dimensionality-upgrading processing on the first intermediate computation result through the network model to obtain the first key-value pair, the method further includes: Obtain first indication information, which indicates whether the first intermediate operation result was generated using the first type of attention mechanism module or the second type of attention mechanism module; The step of performing dimensionality-upgrading processing on the first intermediate computation result through the network model to obtain the first key-value pair includes: If the first indication information indicates that the first intermediate operation result was generated using the first type of attention mechanism module, then the first intermediate operation result is subjected to dimensionality-upgrading processing through the first type of attention mechanism module to obtain the first key-value pair.

3. The method as described in claim 1 or 2, characterized in that, The method further includes: If the first input information does not hit the key-value cache, the hidden state data of the first input information is determined through the network model to obtain the first hidden state data; Based on the first hidden state data, the second key-value pair is determined through the network model; Based on the second key-value pair, the first output result is determined through the network model.

4. The method as described in claim 3, characterized in that, The network model includes a first type of attention mechanism module and a second type of attention mechanism module; the step of determining the second key-value pair based on the first hidden state data through the network model includes: The first hidden state data is reduced in dimensionality using the first type of attention mechanism module to obtain the second intermediate operation result, and the second intermediate operation result is increased in dimensionality using the first type of attention mechanism module to obtain the key-value pairs of the first input information; or, The first hidden state data is processed by the second type of attention mechanism module to obtain the key-value pairs of the first input information.

5. The method as described in claim 4, characterized in that, After performing dimensionality reduction processing on the first hidden state data through the first type of attention mechanism module to obtain the second intermediate calculation result, the method further includes: The result of the second intermediate operation is stored in the key-value cache.

6. The method as described in claim 4 or 5, characterized in that, Before determining the second key-value pair based on the first hidden state data using the network model, the method further includes: Obtain second indication information, which indicates whether to use the first type of attention mechanism module or the second type of attention mechanism module for inference.

7. The method according to any one of claims 4-6, characterized in that, The network model includes multiple attention mechanism modules, each attention mechanism module corresponding to an application domain, and the multiple attention mechanism modules include the first type of attention mechanism module; Before performing dimensionality reduction processing on the first hidden state data through the first type of attention mechanism module, the method further includes: Determine the application domain to which the first input information belongs to obtain the first application domain; The attention mechanism module corresponding to the first application domain among the plurality of attention mechanism modules is identified as the first type of attention mechanism module.

8. The method according to any one of claims 4-7, characterized in that, Before performing dimensionality reduction processing on the first hidden state data through the first type of attention mechanism module, the method further includes: The first type of attention mechanism module is trained online using the second type of attention mechanism module.

9. The method as described in claim 8, characterized in that, The first type of attention mechanism module is the attention mechanism module corresponding to the first application domain; The step of using the second type of attention mechanism module to train the first type of attention mechanism module online includes: When the input information of the second type of attention mechanism module belongs to the first application domain, the first type of attention mechanism module is trained online using the second type of attention mechanism module.

10. The method according to any one of claims 4-9, characterized in that, The second intermediate operation result includes the first key dimensionality reduction result and the first value dimensionality reduction result; The step of performing dimensionality reduction on the first hidden state data to obtain the second intermediate calculation result includes: The first hidden state data is subjected to a first dimensionality reduction process to obtain the first key dimensionality reduction result; The first hidden state data is subjected to a second dimensionality reduction process to obtain the dimensionality reduction result of the first value.

11. The method as described in claim 10, characterized in that, The dimensions of the first key dimensionality reduction result and the first value dimensionality reduction result are different.

12. The method according to any one of claims 1-11, characterized in that, The first intermediate operation result includes the second key dimensionality reduction result and the second value dimensionality reduction result; The step of performing dimensionality-upgrading processing on the first intermediate calculation result to obtain the first key-value pair includes: The second key dimensionality reduction result is then subjected to dimensionality increase processing to obtain the key in the first key-value pair; The dimensionality reduction result of the second value is then subjected to dimensionality upscaling to obtain the value in the first key-value pair.

13. The method according to any one of claims 1-9, characterized in that, The step of performing dimensionality-upgrading processing on the first intermediate calculation result to obtain the first key-value pair includes: The first intermediate operation result is subjected to a first dimensionality increase process to obtain the key in the first key-value pair; The first intermediate operation result is subjected to a second dimensionality increase process to obtain the value in the first key-value pair.

14. The method according to any one of claims 1-13, characterized in that, The dimensionality-upgrading process is performed by a single-layer neural network or a deep neural network.

15. An apparatus for determining the output result of a model, characterized in that, The device includes: The first acquisition module is used to acquire the first input information of the network model; The dimension-upgrading module is used to perform dimension-upgrading processing on the first intermediate operation result through the network model if the first input information hits the first intermediate operation result in the key-value cache, so as to obtain the first key-value pair. The data volume of the first intermediate operation result is less than the data volume of the first key-value pair. The first determining module is used to determine a first output result based on the first key-value pair through the network model, wherein the first output result is the output result corresponding to the first input information.

16. A task execution device, characterized in that, The network model includes a first type of attention mechanism module and a second type of attention mechanism module, and the device includes: The reasoning module is used to perform reasoning tasks using the second type of attention mechanism module to obtain reasoning results; The online training module is used to perform online training tasks on the first type of attention mechanism module using the second type of attention mechanism module. The trained first type of attention mechanism module is used to perform dimensionality reduction processing on the hidden state data of the input information of the network model to obtain intermediate operation results, and to perform dimensionality increase processing on the intermediate operation results to obtain key-value pairs of the input information. The intermediate operation results are used to store in the key-value cache for reuse.

17. A computer device, characterized in that, The computer device includes a processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1-14.

18. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-14.

19. A computer program product, characterized in that, The computer program product stores computer instructions, which, when executed by a processor, implement the method described in any one of claims 1-14.

Images