Large model inference method and related device

Abstract

The present application provides a large model inference method and a related device. The method is applied to a large model. The method comprises: (1) receiving first request information; (2) decoding a first token set and determining a first target token; and (3) determining first response information on the basis of at least one target token and the first target token. The step (2) comprises: on the basis of the first token set, determining a first query vector and N key-value vector sets corresponding to a target layer; on the basis of the first query vector and the N key-value vector sets corresponding to the target layer, determining a first attention score and a first output result corresponding to the first attention score; and determining that the first attention score is greater than or equal to a first preset threshold, and determining the first target token on the basis of the first output result corresponding to the first attention score. The method can reduce the computational cost of each layer of the large model in an inference process, improve inference efficiency, and avoid affecting inference performance.

Description

A method and related equipment for large model inference

[0001] This application claims priority to Chinese Patent Application No. 202510206599.2, filed on February 24, 2025, with the China National Intellectual Property Administration, entitled “A Method and Related Device for Large Model Reasoning”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and more specifically, to a method for large model reasoning, a computing device, a cluster of computing devices, a computer program product, and a computer-readable storage medium. Background Technology

[0003] Currently, the application of large language model (LLM) is becoming increasingly widespread. The steps of LLM inference include: (1) inputting the request information into the LLM model, which divides the request information into at least one initial token; (2) prefilling the at least one initial token to generate a key-value vector for each initial token and obtaining the first generated token; (3) decoding the key-value vector of the at least one initial token and the first generated token to generate a key-value vector for the first generated token and obtaining the second generated token; (4) decoding the key-value vector of the at least one initial token, the key-value vector of the first generated token, and the second generated token to generate a key-value vector for the second generated token and obtaining the third generated token, i.e., repeating step (3) until a preset token is obtained or the number of generated tokens obtained reaches a threshold; (5) outputting response information, which is determined based on at least one generated token. The LLM model includes multiple layers. During pre-filling in the LLM model, each layer obtains and stores the key-value vector of each initial word in at least one initial word corresponding to that layer. During decoding of the at least one initial word and the first generated word, each layer obtains the query vector and key-value vector of the first generated word corresponding to that layer. Each layer uses the query vector of the first generated word and the key-value vector of each initial word in at least one initial word to obtain its output. Each layer stores the key-value vector of the first generated word. The LLM model uses the output of each layer to obtain the second generated word. When the LLM model decodes the at least one initial word, the first generated word, and the second generated word, each layer in the LLM model obtains the query vector and key-value vector of the second generated word corresponding to that layer. Each layer uses the query vector of the second generated word corresponding to that layer, the key-value vector of each initial word in the at least one initial word, and the key-value vector of the first generated word to obtain the output result of that layer. Each layer stores the key-value vector of the second generated word corresponding to that layer. The LLM model uses the output results of each layer to obtain the third generated word. Since each layer of the LLM model needs to use a large number of key-value vectors for calculation during each decoding process, the computational load is large and the computation time is long.Therefore, existing dynamic sparse attention algorithms (DSAes) utilize the key-value vectors of at least one initial word and a portion of the key-value vectors of at least one previously obtained generated word, along with the query vector of the most recently obtained generated word, to obtain the output of each layer of the LLM model during each decoding process, thereby obtaining the next generated word and reducing computational cost. In each decoding process, the number of key-value vectors selected by each layer is the same. However, when the number of key-value vectors used by each layer in each decoding process is small, the accuracy of the generated words is low, resulting in low inference performance of the LLM. Conversely, when the number of key-value vectors used by each layer in each decoding process is large, it is difficult to achieve the goal of reducing computational cost and improving computational efficiency.

[0004] Therefore, how to reduce the computational load in the large model inference process, improve the efficiency of large model inference, and avoid affecting the inference performance of large models has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method, computing device, computing device cluster, computer program product, and computer-readable storage medium for large model inference, which can reduce the amount of computation in the large model inference process, improve the efficiency of large model inference, and avoid affecting the inference performance of large models.

[0006] Firstly, a method for large-scale model inference is provided. This method is applied to a large model and includes: receiving first request information, the first request information including H first initial lexical units, where H is a positive integer; decoding the first lexical unit set to determine a first target lexical unit, the first lexical unit set including at least one first initial lexical unit from the H first initial lexical units and at least one target lexical unit, the at least one target lexical unit being determined based on at least one first initial lexical unit, the large model including at least one layer, the at least one layer including a target layer; and determining first response information based on at least one target lexical unit and the first target lexical unit; wherein, decoding the first lexical unit set to determine the first target lexical unit includes: determining a first query vector corresponding to the target layer and N key-value vector sets based on the first lexical unit set, the N key-value vector sets including at least one... The key-value vector of each initial word in the first initial word set, each of the N key-value vector sets includes at least one key-value vector, and the key-value vector of each initial word set is determined based on at least one initial word set, where N is a positive integer; based on the first query vector corresponding to the target layer and the N key-value vector sets, a first attention score and a first output result corresponding to the first attention score are determined, the first attention score is determined based on the attention score corresponding to each of the M first key-value vector sets, the M first key-value vector sets belong to the N key-value vector sets, M is a positive integer, M≤N; the first attention score is determined to be greater than or equal to a first preset threshold, and the first target word is determined based on the first output result corresponding to the first attention score.

[0007] In this embodiment, after receiving a first request from a user device, the large model processes the first request to obtain a first response. During processing the first request, the large model pre-fills some or all of the initial lexical units included in the first request to obtain a first generated lexical unit and key-value vectors for the corresponding initial lexical units at each layer of the large model. During decoding the initial lexical units and the first generated lexical unit, the large model uses a portion of the key-value vectors from the initial lexical units at each layer and the query vector of the first generated lexical unit to determine a first attention score corresponding to that portion of the key-value vectors. If the first attention score is greater than or equal to a first preset threshold, the model no longer uses more key-value vectors and the query vector of the first generated lexical unit for calculation; instead, it directly uses the output result corresponding to the attention score to obtain the output result of that layer, thereby determining the second generated lexical unit (e.g., the first target lexical unit). In other words, during each decoding process, only a portion of the key-value vectors in the corresponding key-value vectors of each layer of the large model are used, instead of all of them. This reduces the computational load of each layer of the large model during inference, improving inference efficiency. Furthermore, when obtaining generated words (e.g., the first target word) other than the first generated word, in each layer of the large model, the first attention score corresponding to at least one key-value vector used is greater than or equal to a first preset threshold. Therefore, the number of key-value vectors used in each layer is not a fixed value but a variable value. This avoids affecting the accuracy of the obtained generated words, and thus avoids impacting processing performance, even with a smaller number of key-value vectors used in each layer. In other words, the method provided in this embodiment can balance the computational load and accuracy of the large model during inference, thereby minimizing the computational load and improving the accuracy of inference.

[0008] In conjunction with the first aspect, in some implementations, the first attention score is determined by the sum of the attention scores of each of the M first key-value vector sets and a first value, whereby the first value represents the sum of the attention scores of each of the N key-value vector sets.

[0009] In this embodiment, the large model determines the ratio of the sum of attention scores of the calculated key-value vector sets to the sum of attention scores of all key-value vector sets based on the first attention score. This facilitates stopping the calculation of attention scores of other uncalculated key-value vector sets when the first attention score meets a preset threshold, thereby reducing the amount of computation and avoiding affecting the accuracy of the obtained target lexical units.

[0010] In conjunction with the first aspect, in some implementations, the first value is determined by the sum of the attention scores of each of the M first key value vector sets and the second value, the second value is determined by the minimum attention score among the M first key value vector sets and the third value, and the third value is the difference between N and M.

[0011] In this embodiment, the large model uses the minimum attention score among the attention scores of the already calculated key-value vector sets and the number of uncalculated key-value vector sets to simulate the sum of the attention scores of all key-value vector sets, thereby eliminating the need to calculate the sum of the attention scores of all key-value vector sets and reducing the computational load.

[0012] In conjunction with the first aspect, in some implementations, the first importance of each of the M first key-value vector sets is higher than or equal to the first importance of the N key-value vector sets excluding the M first key-value vector sets. The first importance of a key-value vector set is determined based on the metadata vector and the first query vector of the key-value vector set, and the metadata vector of the key-value vector set is determined based on the key-value vector set.

[0013] In this embodiment, the large model sorts the N key-value vector sets according to their first importance from highest to lowest, and then calculates the attention score for each of the sorted N key-value vector sets sequentially. That is, the large model prioritizes calculating the attention score of key-value vector sets with higher first importance, thereby reducing the number of key-value vector sets that need to be calculated when the first attention score reaches a preset threshold, and thus reducing the amount of computation.

[0014] In conjunction with the first aspect, in some implementations, at least one target word includes a second target word, which is determined by pre-filling at least one first initial word; or, at least one target word includes a second target word and a third target word, where the second target word is determined by pre-filling at least one first initial word, and the set of N key-value vectors also includes the key-value vector of the second target word, where the key-value vector of the second target word and the third target word are determined by decoding at least one first initial word and the second target word.

[0015] In this embodiment of the application, after obtaining the first generated word (i.e. the second target word), the large model adds the key value vector of the second target word corresponding to the target layer to the set of N key value vectors, thereby obtaining the second generated word (i.e. the third target word).

[0016] In conjunction with the first aspect, in some implementations, at least one layer of the large model belongs to a set of P layers, and each of the P layer sets includes one or more layers from at least one layer, where P is a positive integer. The method further includes: receiving second request information, the second request information including G second initial lexical units, where G is a positive integer; decoding the first lexical unit set to determine the first target lexical unit, including: obtaining a second output result and the first target lexical unit based on at least one second initial lexical unit from the G second initial lexical units and the first lexical unit set, wherein the second output result is determined by pre-filling at least one second initial lexical unit based on the first layer set, and the first layer set belongs to the P layer sets; or, obtaining a fourth output result and the first target lexical unit based on a third output result and the first lexical unit set, wherein the third output result is the output of the second layer set, and the fourth output result is determined by pre-filling the third output result based on the third layer set, wherein the second layer set and the third layer set belong to different layer sets among the P layer sets.

[0017] In conjunction with the first aspect, in some implementation methods, the fourth target word is determined based on the second or fourth output result; and the second response information is determined based on the fourth target word.

[0018] In this embodiment, the large model performs pre-filling and decoding simultaneously. During each pre-filling process, only one layer from the layer set needs to be used for computation, thus reducing the time required for pre-filling and avoiding the problem of long decoding times due to the long pre-filling time. Furthermore, when at least one second initial lexical unit is G, after performing one pre-filling process using one layer set, the large model can refresh the key-value vectors of the G second initial lexical units corresponding to each layer in the layer set stored in the processing unit's memory to the external memory. That is, the processing unit's memory no longer needs to retain the key-value vectors of the G second initial lexical units corresponding to each layer in the layer set, thereby reducing the memory resource consumption of the processing unit. This external memory is a memory other than the processing unit's memory. The processing unit can read data from this external memory and cache it in its own memory. Compared to external memory, the processing unit reads data from its own memory much faster.

[0019] In conjunction with the first aspect, in some implementations, the large model includes multiple layers, and at least two key-value vectors corresponding to different layers in the multiple layers share a first slot in a first storage space. The first storage space includes at least one slot, and each slot in the at least one slot is used to store at least one key-value vector. The first slot belongs to at least one slot.

[0020] In this embodiment, it is not necessary to divide the first storage space (e.g., memory) into independent blocks according to the layers or processes of the large model; each layer or process of the large model can share the same memory resources. Therefore, memory resource waste can be avoided, and memory resource utilization can be improved.

[0021] Secondly, a computing device is provided. This computing device is used to run a large model. The computing device includes: a transceiver module for receiving first request information, the first request information including H first initial lexical units, where H is a positive integer; a processing module for decoding the first lexical unit set to determine a first target lexical unit, the first lexical unit set including at least one first initial lexical unit from the H first initial lexical units and at least one target lexical unit, the at least one target lexical unit being determined based on at least one first initial lexical unit, the large model including at least one layer, the at least one layer including a target layer; the processing module is further used to determine first response information based on at least one target lexical unit and the first target lexical unit; wherein, the processing module is specifically used to: determine a first query vector corresponding to the target layer and N key-value vector sets based on the first lexical unit set, the N key-value vector sets including at least one first initial lexical unit. Each initial lexical unit has a key-value vector. Each of the N key-value vector sets includes at least one key-value vector. The key-value vector of each initial lexical unit is determined based on at least one initial lexical unit, where N is a positive integer. Based on the first query vector corresponding to the target layer and the N key-value vector sets, a first attention score and a first output result corresponding to the first attention score are determined. The first attention score is determined based on the attention score corresponding to each of the M first key-value vector sets, where the M first key-value vector sets belong to the N key-value vector sets, M is a positive integer, and M≤N. The first attention score is determined to be greater than or equal to a first preset threshold. Based on the first output result corresponding to the first attention score, the first target lexical unit is determined.

[0022] In conjunction with the second aspect, in some implementations, the first attention score is determined by the sum of the attention scores of each of the M first key-value vector sets and a first value, whereby the first value represents the sum of the attention scores of each of the N key-value vector sets.

[0023] In conjunction with the second aspect, in some implementations, the first value is determined by the sum of the attention scores of each of the M first key value vector sets and the second value, the second value is determined by the minimum attention score among the M first key value vector sets and the third value, and the third value is the difference between N and M.

[0024] In conjunction with the second aspect, in some implementations, the first importance of each of the M first key-value vector sets is higher than or equal to the first importance of the N key-value vector sets excluding the M first key-value vector sets. The first importance of a key-value vector set is determined based on the metadata vector and the first query vector of the key-value vector set, and the metadata vector of the key-value vector set is determined based on the key-value vector set.

[0025] In conjunction with the second aspect, in some implementations, at least one target word includes a second target word, which is determined by pre-filling at least one first initial word; or, at least one target word includes a second target word and a third target word, where the second target word is determined by pre-filling at least one first initial word, and the set of N key-value vectors also includes the key-value vector of the second target word, where the key-value vector of the second target word and the third target word are determined by decoding at least one first initial word and the second target word.

[0026] In conjunction with the second aspect, in some implementations, at least one layer of the large model belongs to a set of P layers, and each of the P layer sets includes one or more layers from at least one layer, where P is a positive integer. The transceiver module is also used to receive second request information, which includes G second initial lexical units, where G is a positive integer. The processing module is specifically used to: obtain a second output result and a first target lexical unit based on at least one second initial lexical unit from the G second initial lexical units and a first lexical unit set. The second output result is determined by pre-filling at least one second initial lexical unit from the first layer set, and the first layer set belongs to the P layer sets; or, obtain a fourth output result and a first target lexical unit based on a third output result and a first lexical unit set. The third output result is the output of the second layer set, and the fourth output result is determined by pre-filling the third output result from the third layer set. The second layer set and the third layer set belong to different layer sets among the P layer sets.

[0027] In conjunction with the second aspect, in some implementations, the processing module is also used to: determine the fourth target word based on the second output result or the fourth output result; and determine the second response information based on the fourth target word.

[0028] In conjunction with the second aspect, in some implementations, the large model includes multiple layers, and at least two key-value vectors corresponding to different layers in the multiple layers share a first slot in a first storage space. The first storage space includes at least one slot, and each slot in the at least one slot is used to store at least one key-value vector. The first slot belongs to at least one slot.

[0029] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method in the first aspect or any possible implementation of the first aspect.

[0030] Fourthly, this application provides a computer program product containing instructions that, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method described in the first aspect or any possible implementation thereof.

[0031] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof.

[0032] In a sixth aspect, a chip system is provided, the chip system including logic circuitry for coupling with an input / output interface, through which data is transmitted to perform the method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0033] Figure 1 is a schematic structural diagram of the large model 100.

[0034] Figure 2 is a schematic structural diagram of a cloud scenario.

[0035] Figure 3 is a schematic flowchart of a large model inference method according to an embodiment of this application.

[0036] Figure 4 is a schematic diagram of a method for determining a first attention score according to an embodiment of this application.

[0037] Figure 5 is a schematic flowchart of a method for large model inference according to another embodiment of this application.

[0038] Figure 6 is a schematic diagram of a large model inference method according to an embodiment of this application.

[0039] Figure 7 is a schematic structural diagram of a computing device according to an embodiment of the present application.

[0040] Figure 8 is a schematic structural diagram of a computing device according to an embodiment of the present application.

[0041] Figure 9 is a schematic structural diagram of a computing device cluster according to an embodiment of this application.

[0042] Figure 10 is a schematic diagram of a network connection between computing devices 800A and 800B according to an embodiment of this application. Detailed Implementation

[0043] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0044] This application will present various aspects, embodiments, or features relating to a system comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0045] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0046] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0047] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0048] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0049] To facilitate understanding of the technical solutions in the embodiments of this application, some terms involved in the embodiments of this application are introduced below.

[0050] 1. Token

[0051] In the field of AI, a lexical unit is typically the smallest unit obtained after segmenting text. A lexical unit includes at least one of the following: a number, a letter, a symbol, or a character. For example, a lexical unit may include a word, a subword, at least one letter, at least one number, at least one symbol, or at least one character.

[0052] 2. Large Model

[0053] Large models typically refer to machine learning models with a large number of parameters and complex structures, used in fields such as natural language processing, computer vision, and speech recognition. A typical example of a large model is the LLM (Low-Low Learning Model). An LLM is a deep learning model trained on large amounts of text data. It can generate natural language text or understand the meaning of text, thereby generating corresponding response information based on user input and providing feedback to the user.

[0054] For example, a possible structural diagram of the large model is shown in Figure 1. The large model 100 in Figure 1 includes at least one layer, such as layer 110 and / or layer 120. For example, the large model 100 is an LLM model. Layer 110 and / or layer 120 are transformer layers.

[0055] For example, layer 110 includes a self-attention module 111 and a feed-forward network (FFN) 112. The self-attention module 111 determines at least one query vector and at least one key-value vector based on the input data of layer 110. The input data includes embedding vectors corresponding to the input data of the large model 100, such as embedding vectors corresponding to at least one term included in a request message. Alternatively, the input data includes the output data of the layer preceding layer 110. The layer preceding layer 110 is similar to layer 110, and the way the layer preceding layer obtains its output data is similar to that of layer 110, which will not be described further here. Each key-value vector includes a key vector and a value vector. The self-attention module 111 is also used to perform self-attention computation based on at least one query vector and at least one key-value vector to obtain an attention output. The FFN 112 is used to obtain the output data of layer 110 based on the attention output of the self-attention module 111. The FFN 112 includes two consecutive linear projections with a non-linear activation operation between them.

[0056] For example, suppose the input data of layer 110 is X = [X1, X2, ..., X...]. A A is a positive integer. The self-attention module 111 uses the input data X and the first weight matrix W Q Obtain the query vector Q, Q = W Q ·X. The self-attention module 111 utilizes the input data X and the second weight matrix W K Obtain the key vector K, K = W K ·X. The self-attention module 111 utilizes the input data X and the third weight matrix W V Obtain the value vector V, V = W V ·X. Where the first weight matrix, second weight matrix, and third weight matrix are the weight matrices in layer 110. · represents the dot product. The self-attention module 111 also uses the query vector Q and the key vector K to obtain the attention score vector S, S = (Q·K) T ) / d. Where, K T Let K be the transpose of K, and d be the dimension of the key vector. The self-attention module 111 also uses the attention score vector S to obtain the attention weight P, P = softmax(S). Here, softmax() represents the normalized exponential function. The self-attention module 111 also uses the attention weight P and the value vector V to obtain the attention output O, O = P·V.

[0057] For example, the output data of layer 110 can be used as the input data of the next layer after layer 110. For instance, when the next layer after layer 110 is layer 120, the input data of layer 120 includes the output data of layer 110.

[0058] For example, when layer 120 is the last layer of the large model, the output data of layer 120 is used to determine the output data of the large model. In other words, a generated lexical is determined based on the output data of layer 120. The output data of the large model includes the generated lexical, or at least one lexical adjusted based on the generated lexical.

[0059] 3. Initial Lexical Elements

[0060] After a user inputs information into a large model, the model performs text segmentation on the input to obtain at least one word element included in the input. Each word element in this at least one word element is called an "initial word element." That is, the initial word elements are not word elements generated by the large model, but rather word elements included in the user's input.

[0061] 4. Generate lexical units or target lexical units

[0062] After a user inputs information into a large model, the model generates at least one word based on at least one initial word in the user's input. This generated word is called a "generated word" or "target word." In other words, the generated word or target word is not a word included in the user's input, but rather a word generated by the large model.

[0063] 5. Target layer

[0064] The large model includes at least one layer, which includes a target layer, meaning the target layer is any layer in the large model.

[0065] 6. Prefill treatment

[0066] After receiving user input, the large model pre-fills at least one initial token in the user input to obtain the first generated token. Specifically, the large model utilizes all layers within it to pre-fill the at least one initial token to obtain the first generated token. The implementation of each layer in the large model processing the input data to obtain the output data is described above. The time from inputting at least one initial token into the large model to obtaining the first generated token is called the time-to-first-token (TTFT).

[0067] During the parallel processing of the at least one initial lexical unit, the large model can also obtain the key-value vector of each initial lexical unit in each layer of the large model. The key-value vector of each initial lexical unit corresponding to each layer includes: a key vector and a value vector obtained by the layer after inputting the input data corresponding to the initial lexical unit. When the layer is the first layer of the large model, the input data corresponding to the initial lexical unit includes the embedding vector corresponding to the initial lexical unit. When the layer is a layer in the large model other than the first layer, the input data corresponding to the initial lexical unit includes the output data obtained by the previous layer after processing the input data corresponding to the initial lexical unit.

[0068] For example, the key-value vector of each initial lexical unit in the at least one initial lexical unit corresponding to each layer of the large model is stored in a computing device or storage device so that the large model can use these key-value vectors in subsequent inference processes. The computing device includes a computing device for running the large model, and the storage device is connected to the computing device.

[0069] 7. Decoding Process

[0070] After the large model pre-fills at least one initial lexical unit to obtain the first generated lexical unit, it can determine a set of lexical units to be input. This set of lexical units to be input includes the at least one initial lexical unit and at least one generated lexical unit generated by the large model. For example, after obtaining the (n-1)th generated lexical unit, the set of lexical units to be input includes: at least one initial lexical unit and each generated lexical unit from the first generated lexical unit to the (n-1)th generated lexical unit. Here, n is a positive integer greater than 1.

[0071] The large model decodes the input lexical set to obtain new generated lexical units. For example, if the input lexical set includes at least one initial lexical unit and each of the generated lexical units from the first to the (n-1)th generated lexical unit, the large model decodes the input lexical set to obtain the nth generated lexical unit. Specifically, the large model utilizes all layers within it to decode the input lexical set, thereby obtaining the nth generated lexical unit. The implementation of each layer in the large model processing the input data to obtain the output data is described above. The time from inputting the input lexical set to the large model to obtaining the nth generated lexical unit is called the time-between-token (TBT).

[0072] During the processing of the input lexicon set, the large model can also obtain the query vector and key-value vector of the (n-1)th generated lexicon corresponding to each layer of the large model. This (n-1)th generated lexicon is the most recently obtained generated lexicon by the large model. The query vector of the (n-1)th generated lexicon corresponding to each layer includes: the query vector of the (n-1)th generated lexicon obtained by the layer after inputting the input data corresponding to the (n-1)th generated lexicon. The key-value vector of the (n-1)th generated lexicon corresponding to each layer includes: the key vector and value vector of the (n-1)th generated lexicon obtained by the layer after inputting the input data corresponding to the (n-1)th generated lexicon. When the layer is the first layer of the large model, the input data corresponding to the (n-1)th generated lexicon includes the embedding vector corresponding to the (n-1)th generated lexicon. When the layer is a layer other than the first layer in the large model, the input data corresponding to the (n-1)th generated lexicon includes the output data obtained by the previous layer after processing the input data corresponding to the (n-1)th generated lexicon. When the (n-1)th generated word is processed in each layer of the large model, the large model can also calculate the query vector of the (n-1)th generated word corresponding to the layer and the key value vector of each generated word except the (n-1)th generated word in the set of words to be input corresponding to the layer in each layer, so as to obtain the output data of the layer, and then obtain the nth generated word based on the output data of the layer.

[0073] For example, the key-value vector of the (n-1)th generated lexical corresponding to each layer of the large model is stored in a computing device or storage device so that the large model can use these key-value vectors in subsequent inference processes. The computing device includes a computing device for running the large model, and the storage device is connected to the computing device.

[0074] The method described in this application embodiment can be applied to various computing devices used to run large models. These computing devices include, for example, containers, virtual machines, servers, and computing equipment. When the computing device is a server, the server may include, for example, a local server or a cloud server. When the server is a cloud server, it belongs to the infrastructure managed by a cloud management platform, which is used to provide cloud services to tenants, as shown in Figure 2.

[0075] Figure 2 is a schematic diagram of a cloud service scenario used in this application. As shown in Figure 2, this cloud scenario may include: a cloud management platform 210, the Internet 220, and a client 230. As shown in Figure 2, the cloud management platform 210 is used to manage the infrastructure providing multiple cloud services. The infrastructure includes multiple cloud data centers, each cloud data center includes multiple servers, and each server includes cloud service resources to provide corresponding cloud services to tenants. For example, the cloud service resources may be cloud databases.

[0076] The cloud management platform 210 can be located in a cloud data center. The cloud management platform 210 provides an access interface (such as a user interface or application program interface, API). Tenants can use client 230 to remotely access the access interface to register a cloud account and password on the cloud management platform 210 and log in. After successful authentication of the cloud account and password, the tenant can further select and purchase specific specifications (such as processor, memory, and disk) of computing devices (e.g., containers, virtual machines, servers, computing equipment, etc.) on the cloud management platform 210. After successful purchase, the cloud management platform 210 provides a remote login account and password for the purchased computing device. The tenant can remotely log in to the computing device through client 230 and install and run the tenant's applications on the computing device. Therefore, tenants can create, manage, log in to, and operate computing devices in the cloud data center through the cloud management platform 210. The computing device can also be referred to as a cloud server (Elastic Compute Service, ECS) or elastic instance (different cloud service providers may use different names).

[0077] It should be understood that cloud service tenants can be individuals, businesses, schools, hospitals, government agencies, etc.

[0078] The cloud management platform 210 includes, but is not limited to, a user console, compute management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interaction with tenants. The compute management services manage servers running virtual machines and containers, as well as bare metal servers. The network management services manage network services (such as gateways and firewalls). The storage management services manage storage services (such as data bucket services). The authentication services manage tenant account passwords. The image management services manage virtual machine images. Tenants use client 230 and can log in to the cloud management platform 210 via the internet 220 to manage their rented cloud services.

[0079] Cloud servers are one of the most popular forms of computing resource services today, especially in the era of cloud computing, where they are widely used to host websites, applications, databases, and various other applications. When using cloud servers, it is sometimes necessary to configure multiple Internet Protocol (IP) addresses to extend the services offered by applications. For example, building a cluster of web servers, providing services with different network characteristics, or having different clients access the internet all require configuring multiple IP addresses on the cloud server.

[0080] During the decoding process of a large model, the DSAes method utilizes the key-value vector of at least one initial term corresponding to that layer, a portion of the key-value vectors from the previously obtained n-1 generated terms, and the query vector of the latest obtained nth generated term in each layer of the large model to calculate the output data of that layer, thereby obtaining a new (n+1)th generated term, thus reducing the computational load. n is a positive integer greater than 1. In each decoding process, the number of key-value vectors selected for each layer in the large model is the same. However, when the number of key-value vectors used per layer in each decoding process is small, the accuracy of the obtained generated terms is low, resulting in low inference performance of the large model; when the number of key-value vectors used per layer in each decoding process is large, it is difficult to achieve the goal of reducing computational load and improving computational efficiency. Therefore, this application provides a method for large model inference that can reduce the computational load in the large model inference process, improve the efficiency of large model inference, and avoid affecting the inference performance of the large model.

[0081] Figure 3 is a schematic flowchart of a large model inference method provided in an embodiment of this application. The method in Figure 3 is applied to a large model, which can run on a computing device or a cluster of computing devices, such as any server managed by the cloud management platform 210 in Figure 2. The method in Figure 3 includes the following steps.

[0082] 310, Receive the first request information.

[0083] The large model receives a first request message, which includes H initial lexical units, where H is a positive integer. Each lexical unit includes at least one of the following: a number, a letter, a symbol, or a character. For example, a lexical unit may include a word, a subword, at least one letter, at least one number, at least one symbol, or at least one character.

[0084] In some embodiments, the first request information may be, for example, a question or a description entered by the user.

[0085] In some embodiments, a computing device for running a large model provides a first graphical interface to a user, allowing the user to select or upload first request information within the first graphical interface. The specific form of this first graphical interface is not limited in the embodiments of this application.

[0086] 320. Decode the first set of lexical units to determine the first target lexical unit.

[0087] The large model decodes the first set of lexical units to determine the first target lexical unit. This first set of lexical units includes at least one initial lexical unit from H initial lexical units and at least one target lexical unit, which is determined based on the at least one initial lexical unit. The large model is a pre-trained neural network model. For example, this large model is an LLM model.

[0088] In some embodiments, the large model includes at least one layer, which includes a target layer. Each of the at least one layer is, for example, a transformer layer. Each layer includes a self-attention module and an FFN module, as described in Figure 1.

[0089] In some embodiments, the first lexical set is similar to the lexical set to be input described above.

[0090] For example, the first lexical set includes some or all of the first initial lexical units from H first initial lexical units and a second target lexical unit. The second target lexical unit is determined based on the partial or all of the first initial lexical units.

[0091] For example, by inputting some or all of the H initial lexical units into a large model, a second target lexical unit is obtained, along with the key-value vector of each initial lexical unit in each layer of the large model. The key-value vector of each initial lexical unit corresponding to each layer includes a key vector and a value vector obtained by the layer after inputting the input data corresponding to the initial lexical unit. Wherein, when the layer is the first layer of the large model, the input data corresponding to the initial lexical unit includes the embedding vector corresponding to the initial lexical unit. When the layer is a layer other than the first layer in the large model, the input data corresponding to the initial lexical unit includes the output data obtained by the previous layer after processing the input data corresponding to the initial lexical unit. The second target lexical unit is similar to the first generated lexical unit described above. The first target lexical unit obtained using the set of first lexical units is similar to the n=2nd generated lexical unit described above.

[0092] For example, the first lexical set includes some or all of the H first initial lexical units and a plurality of generated lexical units. The plurality of generated lexical units includes at least a second target lexical unit and a third target lexical unit. The second target lexical unit is determined by pre-filling the partial or all first initial lexical units. The key-value vector of the second target lexical unit and the third target lexical unit are determined by decoding the partial or all first initial lexical units and the second target lexical unit.

[0093] For example, after obtaining the second target lexical unit, some or all of the first initial lexical unit and the second target lexical unit are input into the large model to obtain the third target lexical unit and the key-value vector of the second target lexical unit corresponding to each layer of the large model. The key-value vector of the second target lexical unit corresponding to each layer includes: the key vector and value vector of the second target lexical unit obtained by the layer after inputting the input data corresponding to the second target lexical unit. When the layer is the first layer of the large model, the input data corresponding to the second target lexical unit includes the embedding vector corresponding to the second target lexical unit. When the layer is a layer in the large model other than the first layer, the input data corresponding to the second target lexical unit includes the output data obtained by the previous layer after processing the input data corresponding to the second target lexical unit. The third target lexical unit is similar to the n=2nd generated lexical unit mentioned above. The first target lexical unit obtained using the first lexical unit set is similar to the n=3rd generated lexical unit mentioned above.

[0094] In some embodiments, after obtaining the key-value vectors of the lexical units corresponding to each layer in the large model, the key-value vectors of the lexical units corresponding to each layer are stored in a computing device used to run the large model or in a storage device connected to the computing device to avoid redundant calculations.

[0095] In some embodiments, step 320 includes steps 321-323.

[0096] 321. Based on the first set of word elements, determine the first query vector and the set of N key-value vectors corresponding to the target layer.

[0097] Input the first set of lexical units into the large model to determine the first query vector and N key-value vector sets corresponding to the target layer. These N key-value vector sets include the key-value vector of each of the at least one first initial lexical unit corresponding to that layer. Each of the N key-value vector sets includes at least one key-value vector. The key-value vector of each first initial lexical unit is determined based on that at least one first initial lexical unit, where N is a positive integer.

[0098] In some embodiments, a first set of lexical units is input into a large model. Each layer in the large model processes the input data corresponding to the latest generated lexical unit in the first set of lexical units to obtain the query vector of the latest generated lexical unit corresponding to that layer. The query vector of the latest generated lexical unit corresponding to the target layer is the first query vector.

[0099] For example, after obtaining the second target word, the first query vector becomes the query vector for the second target word corresponding to the target layer. After obtaining the third target word, the first query vector becomes the query vector for the third target word corresponding to the target layer.

[0100] In some embodiments, each layer in the large model determines the key-value vector of each word in the first word set excluding the latest generated word, based on each word in the first word set excluding the latest generated word, thereby determining N key-value vector sets.

[0101] In some embodiments, each of the N key-value vector sets includes the same number of key-value vectors. This application does not limit the value of the number of key-value vectors included in each key-value vector set. Each key-value vector set may also be referred to as a key-value (KV) block.

[0102] In some embodiments, when the first lexical set includes at least one first initial lexical and one generated lexical, the N key-value vector set includes the key-value vector of each first initial lexical in the at least one first initial lexical corresponding to the target layer. When the first lexical set includes at least one first initial lexical and multiple generated lexicals, the N key-value vector set includes: the key-value vector of each initial lexical in the at least one first initial lexical corresponding to the target layer, and the key-value vector of each generated lexical in the multiple generated lexicals corresponding to the target layer, excluding the most recently obtained generated lexical.

[0103] For example, the method by which the large model obtains the query vector and key-value vector for each lexical is described above and will not be repeated here.

[0104] 322. Based on the first query vector corresponding to the target layer and the set of N key-value vectors, determine the first attention score and the first output result corresponding to the first attention score.

[0105] The large model determines the first attention score for the target layer and the first output result corresponding to that first attention score based on the first query vector corresponding to the target layer and N sets of key-value vectors. This first attention score is determined based on the attention score of each of the M sets of first key-value vectors, where the M sets of first key-value vectors belong to the N sets of key-value vectors. Here, M is a positive integer, and M ≤ N.

[0106] Optionally, in the target layer of the large model, based on the first query vector and a set of key-value vectors, the attention score of the key-value vector set and the output result corresponding to the key-value vector set are determined.

[0107] In some embodiments, the attention score of the key-value vector set is determined based on the attention score of each key-value vector in the set. For example, the attention score of the key-value vector set is a weighted sum of the attention scores of each key-value vector in the set. The embodiments of this application do not limit the value of the weight of the attention score of each key-value vector; for example, it can be 1 or other numerical values.

[0108] For example, the attention score of the first key-value vector is determined based on the first query vector and the key vectors within that first key-value vector. This first key-value vector belongs to a set of N key-value vectors. For instance, the attention score of the first key-value vector... Where Q1 represents the first query vector, d1 is the transpose of K1, where K1 is the key vector in the first key value vector and d1 is the dimension of the key vector in the first key value vector.

[0109] In some embodiments, the output corresponding to the key-value vector set is determined based on the attention score of each key-value vector in the set and the value vector within each key-value vector. For example, the output Q corresponding to the key-value vector set. set =P set ·V set Among them, V set This includes the value vector within each key-value vector in the set of key-value vectors. P set =softmax(S set ), S set This includes the attention score for each key-value vector in the set of key-value vectors.

[0110] Optionally, in the target layer of the large model, the first attention score corresponding to the target layer and the first output result corresponding to the first attention score are determined based on the attention score and the corresponding output result of each of the M first key value vector sets.

[0111] In some embodiments, the first attention score corresponding to the target layer is determined by the sum of the attention scores of each of the M first key-value vector sets and a first value, which represents the sum of the attention scores of each of the N key-value vector sets.

[0112] For example, the first attention score corresponding to the target layer is the ratio of the sum of the attention scores of each of the M first key-value vector sets to a first numerical value. That is, the first attention score of the target layer is used to represent the ratio of the sum of the attention scores of the M first key-value vector sets to the sum of the attention scores of the N first key-value vector sets.

[0113] In some embodiments, a first value is determined by the sum of the attention scores of each of the M first key value vector sets and a second value, the second value being determined by the minimum attention score among the M first key value vector sets and a third value, the third value being the difference between N and M.

[0114] For example, the first value is the sum of the attention scores of each of the M first key-value vector sets and the second value, which is the product of the smallest attention score among the M first key-value vector sets and the third value.

[0115] In some embodiments, the first output result corresponding to the first attention score is determined based on the attention score of each of the M first key value vector sets and the output result corresponding to each first key value vector set.

[0116] For example, in the target layer, when the first attention score is determined based on the attention scores of a first set of key-value vectors, the first output result corresponding to the first attention score includes the output result corresponding to the first set of key-value vectors. When the first attention score is determined based on the attention scores of multiple sets of first key-value vectors, i.e., when M is greater than 1, the calculation formula for the first output result corresponding to the first attention score is as follows:

[0117] The first attention score is determined based on the attention scores of the M first key-value vector sets. ES represents the sum of attention scores for each of the M sets of first key-value vectors, from the 1st to the (M-1th)th set. j Let represent the j-th set of first key value vectors in the set of M first key value vectors. ∑ represents the summation symbol. This represents the first output result corresponding to the first attention score, given that the first attention score is determined based on the attention scores of the M-1 sets of first key-value vectors. ES M Let O represent the attention score of the Mth first key-value vector set out of the M sets of first key-value vector sets. M This represents the output result corresponding to the Mth first key value vector set among the M first key value vector sets.

[0118] Optionally, the first importance of each of the M first key-value vector sets is higher than or equal to the first importance of the other key-value vector sets among the N key-value vector sets. The first importance of a key-value vector set is determined based on its metadata vector and the first query vector, and the metadata vector of the key-value vector set is determined based on the key-value vector set itself.

[0119] In some embodiments, the large model obtains a metadata vector for each key-value vector set based on each key-value vector set corresponding to each layer. The metadata vector for each key-value vector set is determined based on each key vector in each key-value vector set.

[0120] For example, the metadata vector for each set of key-value vectors includes the mean of each key vector in the set. For instance, suppose a set of key-value vectors contains c key vectors; the metadata vector for that set is the ratio of a fourth value to c, where c is a positive integer. This fourth value is the ratio of the sum of the elements of each key vector in the set to the dimension of that key vector. Alternatively, the metadata vector for each set of key-value vectors includes the bounding cuboid of that set.

[0121] In some embodiments, in the target layer of the large model, the first importance of each of the N key-value vector sets is determined based on a first query vector and the metadata vector of each of the N key-value vector sets. Exemplarily, the first importance of each key-value vector set includes the dot product of the first query vector and the metadata vector of that key-value vector set.

[0122] In some embodiments, in the target layer of a large model, after determining the first importance of each of the N key-value vector sets, the N key-value vector sets are sorted in descending order of their first importance. Among the sorted N key-value vector sets, the first importance of the first key-value vector set is greater than or equal to the first importance of all other key-value vector sets in the N sets except for the first key-value vector set, and the first importance of the Nth key-value vector set is less than or equal to the first importance of all other key-value vector sets in the N sets except for the Nth key-value vector set.

[0123] In some embodiments, in the target layer of the large model, the first attention score and the corresponding first output result are obtained by sequentially calculating the first attention score of the target layer based on the sorted N key-value vector sets and the first query vector. In other words, in the target layer of the large model, the attention score and corresponding output result of the first key-value vector set are first calculated based on the first key-value vector set in the sorted N key-value vector sets and the first query vector. The large model uses the attention score of the first key-value vector set as the first attention score and determines whether the first attention score is greater than or equal to a first preset threshold, i.e., proceeding to step 323. When the first attention score is less than the first preset threshold, the attention score and corresponding output result of the second key-value vector set are calculated based on the second key-value vector set in the sorted N key-value vector sets and the first query vector. The large model updates the first attention score, and the updated first attention score is the sum of the first attention score before the update and the attention score of the second key-value vector set. The large model determines whether the updated first attention score is greater than or equal to the first preset threshold, i.e., proceeding to step 323. Repeat the above steps in the target layer of the large model until the updated first attention score is greater than or equal to the first preset threshold.

[0124] 323, determine that the first attention score is greater than or equal to the first preset threshold, and determine the first target word based on the first output result corresponding to the first attention score.

[0125] After determining the first attention score, the large model determines whether the first attention score is greater than or equal to a first preset threshold. This first preset threshold is a positive number less than or equal to 1. The specific value of the first preset threshold is not limited in this application embodiment; for example, it can be 0.8, 0.85, 0.9, 0.95, etc.

[0126] In some embodiments, when the first attention score is greater than or equal to a first preset threshold, the large model uses the first output result corresponding to the first attention score as the output result of the self-attention module of the target layer, and calculates the first output result based on the FFN in the target layer to obtain the output result of the target layer. The large model determines the first target word based on the output result of the target layer.

[0127] For example, when the first attention score is greater than or equal to a first preset threshold, and the target layer is a layer other than the last layer in the large model, the output of the target layer is used as the input data for the layer following the target layer to obtain the first target word. When the target layer is the last layer in the large model, the output of the target layer is processed to determine the first target word. For example, the output of the target layer is subjected to linear transformation, softmax function processing, and word generation steps to determine the first target word. The specific implementation methods of these linear transformation, softmax normalization processing, and word generation steps are as described in the implementation methods of LLM model word generation in the prior art, and will not be elaborated here.

[0128] In some embodiments, if the first attention score is less than or equal to a first preset threshold, the large model determines the (M+1)th key-value vector set corresponding to the target layer. This (M+1)th key-value vector set belongs to the N key-value vector sets. Based on the first query vector corresponding to the target layer and the (M+1)th key-value vector set, the large model updates the first attention score and the first output result corresponding to the first attention score, and determines whether the updated first attention score is greater than or equal to the first preset threshold. The large model repeats the above steps in the target layer until the updated first attention score is greater than or equal to the first preset threshold.

[0129] For example, the large model determines the attention score and corresponding output result of the (M+1)th key-value vector set based on the first query vector corresponding to the target layer and the (M+1)th key-value vector set. The large model updates the first attention score, and the updated attention score is the sum of the attention score before the update and the attention score of the (M+1)th key-value vector set. The large model updates the first output result corresponding to the first attention score, and the updated first output result is determined according to the above formulas (1) and (2).

[0130] For example, the first importance of the (M+1)th key-value vector set is less than or equal to the first importance of the Mth key-value vector set.

[0131] In some embodiments, the large model repeats step 320 until the obtained generated lexical is a preset lexical, or the number of obtained generated lexicals is greater than or equal to a third preset threshold. In other words, after obtaining the first target lexical, the large model determines whether the first target lexical is a preset lexical, and / or determines whether the number of obtained generated lexicals is greater than or equal to the third preset threshold. The specific value of the third preset threshold is not limited in the embodiments of this application. When the first target lexical is a preset lexical, and / or the number of obtained generated lexicals is greater than or equal to the third preset threshold, the computing device executes step 330. When the first target lexical is not a preset lexical, and / or the number of obtained generated lexicals is less than the third preset threshold, the computing device repeats step 320.

[0132] For example, preset morphemes include, for instance, the end of sentence (EOS) morpheme.

[0133] In some embodiments, when the first attention score corresponding to each layer of a large model is greater than or equal to a first preset threshold, the number of the first key-value vector sets used to determine the first attention score of each layer may be the same or different.

[0134] 330. Determine the first response information based on at least one target word and the first target word.

[0135] After obtaining at least one target word and a first target word from the first word set, the large model determines first response information. This first response information includes the at least one target word and the first target word. For example, the first response information includes a concatenation of the at least one target word and the first target word. Alternatively, the first response information is obtained after adjusting the at least one target word and the first target word. For example, the first response information includes at least one target word and some or all of the words in the first target word set, as well as at least one word other than the at least one target word and the first target word; or, the first response information includes at least one target word and some or all of the words in the first target word set.

[0136] In some embodiments, the computing device for running a large model provides a second graphical interface to the user, allowing the user to view the first response information in the second graphical interface. The embodiments of this application do not limit the specific form of this second graphical interface.

[0137] Optionally, if the first lexical set includes some of the H first initial lexical units, the large model can also determine a second lexical set. This second lexical set includes some of the H first initial lexical units. At least one first initial lexical unit included in the second lexical set is different from at least one first initial lexical unit included in the first lexical set. That is, the position of each first initial lexical unit included in the second lexical set among the H first initial lexical units is different from the position of each first initial lexical unit included in the first lexical set among the H first initial lexical units. The large model performs pre-filling and decoding processing based on the second lexical set to determine at least one target lexical unit corresponding to the second lexical set. The specific implementation is described in step 320 and will not be repeated here. The large model determines the first response information based on at least one target lexical unit corresponding to the second lexical set and at least one target lexical unit corresponding to the first lexical set. The specific implementation is similar to step 330 and will not be repeated here.

[0138] Optionally, an algorithm example for executing step 322 in the target layer of a large model is shown in Figure 4. Figure 4 is a schematic diagram of the method for determining the first attention score corresponding to the target layer provided in the embodiments of this application. Figure 4 includes 18 lines of code. The first line of code in Figure 4 indicates that step 322 is started based on parameters q and B, where parameter q represents the first query vector corresponding to the target layer, and parameter B includes a set of N key-value vectors corresponding to the target layer. The second line of code in Figure 4 indicates setting vector O. acc =[0,…,0] d d is a vector O acc The dimension of vector O is d, where d is a preset value. acc This is used to represent the first output result corresponding to the first attention score. The value of d has the same dimension as the key vector in each of the N key vector sets. The third line of code in Figure 4 is used to set the variable ES. acc =0, variable ES acc This is used to represent the first attention score. Line 4 of the code in Figure 4 is used to set the variable ES. min =MAX_VAL, where MAX_VAL is a preset value that is as large as possible. Line 5 of Figure 4 indicates that the variable BS is determined using the first query vector q and the metadata vector set metadata. The metadata vector set metadata includes the metadata vector for each of the N key-value vector sets. The variable BS includes the first importance of each of the N key-value vector sets. Line 6 of Figure 4 indicates that each key-value vector set in parameter B is sorted according to its first importance from highest to lowest to obtain B'. B' includes the sorted N key-value vector sets. Line 7 of Figure 4 indicates that the variable N is set...left N represents the number of key-value vector sets corresponding to the target layer. left =N. Line 8 of the code in Figure 4 indicates that lines 9-16 are executed sequentially for each of the sorted N key-value vector sets. Line 9 of the code in Figure 4 indicates that the parameter KV is set. bi Including the i-th key-value vector set b in the sorted set of N key-value vectors i , i = 1, ..., N. Line 10 of the code in Figure 4 is used to represent the expression based on parameters q and KV. bi Calculate the attention score ES for the i-th key-value vector set and the output result O corresponding to the i-th key-value vector set. Line 11 of the code in Figure 4 represents the calculation of the output result O corresponding to the first i key-value vector sets. acc O acc =(ES) acc ·O acc +ES·O) / (ES acc +ES). Line 12 of the code in Figure 4 is used to calculate the sum of attention scores ES for the first i key-value vector sets. acc ES acc =ES+ES acc Line 13 of the code in Figure 4 is used to determine the minimum attention score ES for the first i key-value vector sets. min ES min =MIN(ES,ES) min ), where MIN represents the function to find the minimum value. Line 14 of the code in Figure 4 is used to update the variable N. left The updated variable N left =N left -1. Line 15 of the code in Figure 4 is used to calculate the first attention score P. acc P acc =ES acc / (ES acc +ES min ·N left Line 16 of the code in Figure 4 is used to indicate determining P. acc Is it greater than the first preset threshold ε? In P acc If the value exceeds the first preset threshold ε, lines 17 and 18 of the code in Figure 4 are executed. Line 17 of the code in Figure 4 indicates that the loop stops, meaning that lines 9-17 are no longer executed sequentially on the key-value vector sets following the i-th key-value vector set in the sorted N key-value vector sets. Line 18 of the code in Figure 4 indicates that the output is O. acc O acc This is the first output result corresponding to the first attention score. In P accIf the value is less than or equal to the first preset threshold ε, line 17 of the code in Figure 4 will not be executed, and lines 9-16 will be executed sequentially for the key-value vector set after the (i+1)th key-value vector set in the sorted N key-value vector sets.

[0139] In some embodiments, line 10 of the code in Figure 4 needs to call the attention kernel function in the target layer of the large model at runtime. However, frequent calls to the attention kernel function can easily lead to significant resource overhead. Therefore, when running line 9 of the code in Figure 4, the parameter KV can be set. bi Including the i-th key-value vector set b in the sorted set of N key-value vectors i up to the (i+g)th key-value vector set b i+g Each set of key-value vectors in the attention kernel is used to calculate the attention score and corresponding output of the set of g key-value vectors with each call to the attention kernel function, thereby reducing the number of calls to the attention kernel function and reducing resource overhead. Where i + g = 2, ..., N, g is a positive integer and a preset value. The specific value of g is not limited in this embodiment.

[0140] In some embodiments, in the 18 lines of code in Figure 4, line 10 is executed by the graphics processing unit (GPU), while the remaining lines are executed by the central processing unit (CPU). When the GPU executes line 10, it includes the following steps: loading a set of key-value vectors not currently stored in GPU memory; and calculating the attention score and corresponding output of the key-value vector set. Since loading the key-value vector set sequentially is relatively quick, it can easily lead to prolonged idle time for the GPU resources responsible for this step. Therefore, large models can use independent compute unified device architecture (CUDA) pipelines to overlap and execute these steps in a pipelined manner. For example, while some GPU resources load the key-value vector set, other GPU resources are used to calculate the attention score and corresponding output of the already loaded key-value vector set.

[0141] For example, the large model asynchronously checks whether the first attention score is greater than or equal to a first preset threshold by monitoring a thread, thereby reducing the synchronization overhead between the CPU and GPU. That is, while the GPU calculates the attention score and corresponding output of the key-value vector set, the CPU asynchronously calculates the first attention score and confirms whether the first attention score is greater than or equal to the first preset threshold. When the first attention score is greater than or equal to the first preset threshold, the CPU notifies the GPU to stop calculating the attention score and corresponding output of the key-value vector set corresponding to the target layer.

[0142] In the method shown in Figure 3, during each decoding process, only a portion of the key-value vectors in each layer of the large model are used, rather than all of them. This reduces the computational load of each layer in the large model during inference, improving the efficiency of the large model's inference. Furthermore, when obtaining other generated words (e.g., the first target word) besides the first generated word, the first attention score corresponding to at least one key-value vector used in each layer of the large model is greater than or equal to a first preset threshold. Therefore, the number of key-value vectors used in each layer is not a fixed value but a variable value. This avoids affecting the accuracy of the obtained generated words and thus the inference performance of the large model when the number of key-value vectors used in each layer is relatively small.

[0143] In some embodiments, the pre-filling process, where at least one initial lexical unit is processed in parallel during a single iteration of the large model, achieves efficient utilization of the processing unit's computational resources. In contrast, the decoding process involves a complete forward propagation of the most recently acquired generated lexical unit, resulting in low computational utilization and memory constraints. To improve the utilization of the processing unit, a hybrid batching method combines the pre-filling and decoding phases of different requests into a single batch. That is, during a single run of the large model, some or all of the initial lexical units in request A are pre-filled, while simultaneously decoding some or all of the initial lexical units and at least one generated lexical unit corresponding to request B, which has already undergone pre-filling. However, because the pre-filling process requires parallel processing of a larger number of initial lexical units corresponding to request A, the processing time required for pre-filling is significantly higher than that required for decoding, resulting in a larger TBT (Total Time Limit) for request B.

[0144] In some embodiments, the large model may execute the method in Figure 5 while executing the method in Figure 3. Alternatively, the large model may execute the method in Figure 5 independently. In other words, the methods in Figure 3 and Figure 5 may be used in combination, or the methods in Figure 3 and Figure 5 may be used independently.

[0145] Figure 5 is a schematic flowchart of a large model inference method provided in an embodiment of this application. The method in Figure 5 is applied to a large model, which can run on a computing device or a cluster of computing devices, such as any server managed by the cloud management platform 210 in Figure 2. The method in Figure 5 includes the following steps.

[0146] 510, Receive the second request information.

[0147] The large model receives a second request message, which includes G second initial lexical units, where G is a positive integer. These lexical units are described in step 310.

[0148] In some embodiments, the second request information may be, for example, a question or a description entered by the user.

[0149] In some embodiments, the computing device for running large models provides a third graphical interface to the user, allowing the user to select or upload second request information within the third graphical interface. The specific form of this third graphical interface is not limited in the embodiments of this application.

[0150] 520. Based on at least one of the G second initial lexical units and the first lexical unit set, obtain the second output result and the first target lexical unit.

[0151] Input at least one of the G second initial lexical units and the first lexical unit set into the large model simultaneously to obtain the output of the large model, which includes the second output result and the first target lexical unit.

[0152] In some embodiments, the large model is described in Figure 1 or Figure 3. The first lexical set and the first target lexical are described in Figure 3.

[0153] In some embodiments, at least one layer of the large model belongs to P sets of layers, each of the P sets of layers including one or more layers from the at least one layer. P is a positive integer, and P is a preset value.

[0154] For example, the first layer in each of the P layer sets is connected to the last layer in the layer set preceding it, meaning the output data of the last layer in the layer set preceding it is the input data of the first layer in that layer set. Similarly, the last layer in each of the P layer sets is connected to the first layer in the layer set following it, meaning the output data of the last layer in that layer set is the input data of the first layer in the layer set following it.

[0155] For example, when each of the P layer sets includes multiple layers, these multiple layers are connected sequentially. That is, the output data of the first layer in each layer set is the input data of the second layer in that layer set, the output data of the second layer in that layer set is the input data of the third layer in that layer set, and so on.

[0156] For example, each of the P layer sets includes the same number of layers.

[0157] For example, the number of layers included in each of the P layer sets is determined based on at least one of the following: the length of the request message, the service level objective (SLO) of TBT, the batch size of the decoding request, and the system decoding throughput. The length of the request message refers to the number of initial tokens included in the request message. The TBT SLO is, for example, the TBT obtained from a single decoding process on the large model, or the average of the TBTs obtained from multiple decoding processes on the large model. The system decoding throughput = (1 / TBT ​​SLO) * batch size of the decoding request. Multiple decoding requests form a batch, and when this batch is input together into the large model for processing, the number of requests in the batch is the batch size. This decoding request is used to request the large model to decode at least one initial token and at least one target token to generate a new target token.

[0158] For example, the fewer layers included in each layer set, the lower the TBT and the higher the system decoding throughput, but the efficiency of pre-filling at least one second initial token to obtain the fourth target token is lower.

[0159] For example, the number of layers included in each layer set can be initially set to a small value, such as 1. After at least one pre-padding process and at least one decoding process using the large model, it is determined whether at least one of the following is satisfied: whether TBT is greater than or equal to the TBT threshold, and whether the system decoding throughput is greater than or equal to the throughput threshold. If TBT is less than the TBT threshold, and / or the system decoding throughput is less than the throughput threshold, the number of layers in each layer set is updated, for example, to 2. The above operation is repeated until TBT is greater than or equal to the TBT threshold, and / or until the system decoding throughput is greater than or equal to the throughput threshold.

[0160] For example, when setting a specific value for the number of layers included in each layer set, a first functional relationship between the length of the request information and the number of layers in the layer set can be determined, and / or a second functional relationship between the batch size of the decoding request and the number of layers in the layer set can be determined, thereby determining the number of layers in the layer set based on the length of the request information and / or the batch size of the decoding request. This application embodiment does not limit the specific form of the above-mentioned first or second functional relationship.

[0161] For example, the first functional relationship includes: Similarly, the second functional relationship includes: in, This indicates rounding up. This configurable value is adjustable via TBT and / or system decoding throughput. It should be understood that this rounding up can be replaced by rounding down, rounding to the nearest integer, or other rounding methods.

[0162] In some embodiments, the large model pre-fills at least one of the G second initial tokens using a first layer set to obtain a second output result. This first layer set belongs to the P layer sets. In other words, after simultaneously inputting at least one of the G second initial tokens and the first token set into the large model, each layer in the first layer set of the large model sequentially pre-fills the at least one second initial token to obtain the output result of the last layer in the first layer set, i.e., the second output result.

[0163] In some embodiments, the large model decodes the first word set to obtain the first target word. In other words, after simultaneously inputting at least one of the G second initial word sets and the first word set into the large model, each layer in the large model sequentially decodes the first word set to obtain the first target word.

[0164] For example, suppose the large model includes three layers (layer 610, layer 620, and layer 630), which belong to three layer sets, i.e., each layer set includes one layer. Step 520 is implemented as shown in Figure 6(a). During a single run of the large model, at least one second initial lexical and a lexical set 601 are simultaneously input into the first layer (layer 610) of the large model. Layer 610 pre-fills at least one second initial lexical, obtaining output 611. Layer 610 decodes the lexical set 601, obtaining output 612. Output 612 is input into the next layer (layer 620) of layer 610, where it is decoded to obtain output 621. Output 621 is input into the next layer (layer 630) of layer 620, where it is decoded to obtain output 631. With the lexical set 601 being the first lexical set, the large model 600 obtains the first target lexical based on the output result 631. The output result 611 is the second output result.

[0165] For example, the first layer set is the first layer set among P layer sets.

[0166] In some embodiments, after inputting at least one of the G second initial lexical units into the first layer set, the large model can also obtain the key-value vector of each second initial lexical unit in the at least one second initial lexical unit corresponding to each layer in the first layer set. The key-value vector of each second initial lexical unit corresponding to each layer has a similar meaning to the key-value vector corresponding to each first initial lexical unit corresponding to each layer.

[0167] In some embodiments, the key-value vector of each of the at least one second initial lexical corresponding to each layer in the first layer set is stored in a computing device for running the large model or in a storage device connected to the computing device to avoid redundant calculations.

[0168] Optionally, step 520 can be replaced by step 530, that is, either step 520 or step 530 can be performed.

[0169] 530. Based on the third output result and the first word set, obtain the fourth output result and the first target word.

[0170] The third output and the first word set are simultaneously input into the large model to obtain the output of the large model, which includes the fourth output and the first target word.

[0171] In some embodiments, the large model is described in Figure 1 or Figure 3. The first lexical set and the first target lexical are described in Figure 3.

[0172] In some embodiments, the third output is the output of the second layer set, which belongs to P layer sets. The P layer sets are described in step 520. The method for obtaining the third output is similar to the method for obtaining the fourth output, and will not be repeated here.

[0173] In some embodiments, the large model pre-fills the third output result using a third layer set to obtain a fourth output result. This third layer set belongs to the layer set following the second layer set in the P layer sets. In other words, after the third output result and the first word set are simultaneously input into the large model, each layer in the third layer set of the large model sequentially pre-fills the third output result to obtain the output result of the last layer in the third layer set, i.e., the fourth output result.

[0174] In some embodiments, the large model decodes the first word set to obtain the first target word. In other words, after simultaneously inputting at least one of the G second initial word sets and the first word set into the large model, each layer in the large model sequentially decodes the first word set to obtain the first target word.

[0175] For example, suppose the large model includes three layers (layer 610, layer 620, and layer 630), which belong to three layer sets, meaning each layer set includes one layer. Step 530 is implemented as shown in Figure 6(b) or (c). As shown in Figure 6(b), when the large model is run again, output 611 is input into layer 620, which pre-fills output 611 to obtain output 622. Lexical set 602 is input into layer 610, which decodes lexical set 602 to obtain output 613. Output 613 is input into the next layer after layer 610 (i.e., layer 620), which decodes output 613 to obtain output 623. Output 623 is input into the next layer after layer 620 (i.e., layer 630), which decodes output 623 to obtain output 632. When the lexical set 602 is the first lexical set, the large model 600 obtains the first target lexical based on the output result 632. Output result 611 is the third output result, and output result 622 is the fourth output result. As shown in Figure 6(c), when the large model is run again, output result 622 is input into layer 630, and layer 630 performs pre-filling processing on output result 622 to obtain output result 633. Lexical set 603 is input into layer 610, and layer 610 performs decoding processing on lexical set 603 to obtain output result 614 of layer 610. Output result 614 is input into the next layer after layer 610 (i.e., layer 620), and layer 620 performs decoding processing on output result 614 to obtain output result 624. Output result 624 is input into the next layer after layer 620 (i.e., layer 630), and layer 630 performs decoding processing on output result 624 to obtain output result 634. With lexical set 603 as the first lexical set, the large model 600 obtains the first target lexical based on output 634. Output 622 is the third output, and output 633 is the fourth output.

[0176] For example, lexical set 602 includes lexical set 601 and target lexical units obtained from output result 631. Lexical set 603 includes lexical set 602 and target lexical units obtained from output result 632.

[0177] For example, the implementation method of pre-filling each layer of the large model to obtain the output result in the embodiments of this application is not limited. The methods of pre-filling the input data to obtain the output data in each layer of the LLM model in the prior art can all be applied to the embodiments of this application.

[0178] For example, the implementation method of obtaining the first target lexical unit in the embodiments of this application is not limited. For example, the first target lexical unit can be obtained by pre-filling the LLM model in the prior art, and the first target lexical unit is the nth generated lexical unit. Alternatively, the first target lexical unit can be obtained according to step 320 in Figure 3. Alternatively, the first target lexical unit can be obtained according to the DSAes method.

[0179] In some embodiments, after inputting the third output result into the third layer set, the large model can also obtain the key-value vector of each second initial word in the at least one second initial word corresponding to each layer in the third layer set. The key-value vector of each second initial word corresponding to each layer has a similar meaning to the key-value vector corresponding to each first initial word corresponding to each layer.

[0180] In some embodiments, the key-value vector of each of the at least one second initial lexical corresponding to each layer in the third layer set is stored in a computing device for running the large model or in a storage device connected to the computing device to avoid redundant calculations.

[0181] 540. Determine the fourth target word based on the second or fourth output result.

[0182] With P=1, the last layer in the first layer set is the last layer of the large model. The large model determines the fourth target word based on the second output result.

[0183] When P is greater than 1, the first layer set is the first layer set among P layer sets. The P layer sets also include other layer sets besides the first layer set, such as the third layer set. When the third layer set is the last layer set among the P layer sets, the last layer in the third layer set is the last layer of the large model. The large model determines the fourth target word based on the fourth output result. Alternatively, when the third layer set is not the last layer set among the P layer sets, the P layer sets still include other layer sets besides the first and third layer sets, such as the last layer set. The last layer in this last layer set is the last layer of the large model. The large model determines the fourth target word based on the output result of this last layer set. The implementation method for the large model to obtain the output result of this last layer set is similar to the implementation method for obtaining the fourth output result, and will not be repeated here.

[0184] For example, as shown in Figure 6, the computing device determines the fourth target lexical unit based on the output result 633. For instance, the large model performs linear transformation, softmax normalization, and lexical unit generation on the output result 633 to determine the fourth target lexical unit. The specific implementation methods of these linear transformation, softmax function processing, and lexical unit generation steps are described in the prior art when LLM models generate lexical units, and will not be elaborated here. This fourth target lexical unit is similar to the first generated lexical unit described above.

[0185] 550. Based on the fourth target word, determine the second response information.

[0186] After obtaining the fourth target word, the large model determines the second response information based on the fourth target word. This second response information includes the fourth target word. Alternatively, the second response information includes at least one target word obtained based on the fourth target word. The specific implementation is similar to step 330.

[0187] In some embodiments, the large model determines a third lexical set. This third lexical set includes at least one of the G second initial lexical units and a fourth target lexical unit. The large model decodes the third lexical set to obtain a fifth target lexical unit. Based on the third lexical set and the fifth target lexical unit, the large model determines the second response information, in a manner similar to step 330.

[0188] For example, the embodiments of this application do not limit the implementation method of obtaining the fifth target word in the large model. For example, the fifth target word can be obtained by decoding the LLM model in the prior art, and the fifth target word is the nth generated word. Alternatively, the fifth target word can be obtained according to step 320 in Figure 3. Alternatively, the fifth target word can be obtained according to the DSAes method.

[0189] In some embodiments, the computing device for running a large model provides a fourth graphical interface to the user, allowing the user to view the second response information in the fourth graphical interface. The embodiments of this application do not limit the specific form of this fourth graphical interface.

[0190] Optionally, if the third word set includes some of the G second initial word units, the large model can also determine a fourth word set. This fourth word set includes some of the G second initial word units. At least one second initial word unit in this fourth word set is different from at least one second initial word unit in the third word set. That is, the position of each second initial word unit in the fourth word set among the G second initial word units is different from the position of each second initial word unit in the third word set among the G second initial word units. The large model performs pre-filling and decoding processing based on the fourth word set to determine at least one target word unit corresponding to the fourth word set. The specific implementation is described in step 320 and will not be repeated here. The large model determines the second response information based on at least one target word unit corresponding to the fourth word set and at least one target word unit corresponding to the third word set. The specific implementation is similar to step 330 and will not be repeated here.

[0191] In the method shown in Figure 5, although the large model performs pre-filling and decoding simultaneously, each pre-filling operation only requires computation using layers from one layer set. This reduces the time required for pre-filling, avoiding the problem of long decoding times due to longer pre-filling times, thus reducing TBT (Time To Be Calculated). Furthermore, when at least one second initial lexical unit in step 520 is G second initial lexical units, after performing one pre-filling operation using one layer set, the large model can refresh the key-value vectors of the G second initial lexical units corresponding to each layer in that layer set, already stored in the processing unit's (e.g., GPU) memory, to external memory. This means the processing unit no longer needs to retain the key-value vectors of the G second initial lexical units corresponding to each layer in that layer set in its memory, thereby reducing the memory resource consumption of the processing unit. This external memory is memory other than the processing unit's memory. The processing unit can read data from this external memory and cache it in its own memory. Compared to external memory, the processing unit reads data from its own memory much faster.

[0192] Optionally, since the self-attention module in at least one layer of the large model needs to perform calculations using the key-value vectors of each initial word corresponding to that layer and the key-value vectors of each previously obtained generated word corresponding to that layer during the process of obtaining generated words, the key-value vectors of each initial word corresponding to each layer and the key-value vectors of each previously obtained generated word corresponding to each layer are usually cached in the processing unit memory to avoid redundant calculations. This processing unit includes, for example, a CPU, GPU, or neural network processing unit (NPU). At least one key-value vector in this cache is called a key-value (KV) cache. Since the size of the KV cache increases as the decoding process progresses, the processing unit memory is usually divided into fixed-size blocks, and the KV cache is stored in multiple non-contiguous memory blocks to avoid memory fragmentation. For example, paged attention can be used to divide the memory into multiple non-contiguous blocks.

[0193] In some embodiments, when running a large model, the memory management method of the processing unit includes: dividing the memory of the processing unit into multiple independent blocks according to the processing procedure of the large model, so that the large model can only use the memory space corresponding to each processing procedure. Alternatively, dividing the memory of the processing unit into multiple independent blocks according to the layers of the large model, so that the large model can only use the memory space corresponding to each layer during the processing of each layer. However, the above methods have some problems. For example, when executing the method in Figure 3, since the number of key-value vector sets used by each layer of the large model is different, it is difficult to determine the size of the memory space corresponding to each layer, and it is easy to cause the problem that the storage resources of some layers are insufficient while the storage resources of some layers are wasted. When executing the method in Figure 5, the large model performs pre-filling processing and decoding processing simultaneously. Since the pre-filling processing needs to store a large number of newly obtained key-value vector sets, it requires a large amount of memory space, and the decoding processing needs to utilize a large number of already stored key-value vector sets, it is easy to cause significant interference to the decoding process. In view of this, the embodiments of this application provide a memory management method in the inference process of a large model. This method can be used in conjunction with the methods in Figures 3 and / or 5, or it can be used alone; this embodiment of the application is not limited in this regard. Specifically, the large model manages the memory (e.g., GPU memory) of the processing units used to run the large model through a global tensor. The large model divides the memory used by the processing units into at least one slot, each slot for storing at least one set of key-value vectors. During the operation of the large model, each layer in the large model can utilize this at least one slot to store the set of key-value vectors during pre-filling or decoding processes. In other words, it is not necessary to divide the memory into independent blocks according to the layers or processes of the large model; each layer and / or each process of the large model shares the same memory resources. Therefore, memory resource waste can be avoided, and memory resource utilization can be improved.

[0194] In some embodiments, the large model includes multiple layers, and at least two key-value vectors corresponding to different layers share a first slot in a first storage space. The first storage space includes at least one slot, each of which is used to store multiple key-value vectors, and the first slot belongs to at least one slot. Exemplarily, the first storage space is the memory of a processing unit.

[0195] For example, each of the at least one slot includes contiguous storage space, and the at least one slot is a contiguous slot. In other words, when the second slot is adjacent to the first slot, the last storage location in the first slot is adjacent to the first storage location in the second slot.

[0196] In some embodiments, the large model includes a KV cache manager. This KV cache manager is used to manage slots in the first storage space; for example, the KV cache manager is responsible for allocating key-value vectors, reclaiming key-value vectors, and loading key-value vectors for each slot.

[0197] In some embodiments, each slot in the first storage space has attributes including at least one of the following: free, pre-filled, and decoded. When no key-value vector is stored in the first slot, the attribute of the first slot is free. When a key-value vector obtained during the pre-filling process is stored in the first slot, the attribute of the first slot is pre-filled. When a key-value vector required for the decoding process is stored in the first slot, the attribute of the first slot is decoded.

[0198] In some embodiments, when storing a key-value vector in a first storage space, the KV cache manager determines whether the first storage space includes slots with the attribute of being free. If the first storage space includes slots with the attribute of being free, the KV cache manager stores the key-value vector in the free slot. If the first storage space does not include slots with the attribute of being free, the KV cache manager first determines a slot from the slots with the attribute of being pre-filled, evicts the data in that slot, and stores the key-value vector in that slot. If the slots with the attribute of being pre-filled do not include slots where data can be evicted, the KV cache manager determines a slot from the slots with the attribute of being decoded, evicts the data in that slot, and stores the key-value vector in that slot.

[0199] In some embodiments, the method in Figure 5 ensures that the key-value vector corresponding to each layer in the set of layers of the large model is accessed only once during the pre-filling process, and the method in Figure 3 ensures that each layer of the large model only needs to use a portion of the key-value vector corresponding to each layer during the decoding process. Therefore, prioritizing the determination of the slots for evicted data from the slots with the attribute of pre-filling can minimize the need to evict key-value vectors that will need to be used again in a short period of time.

[0200] In some embodiments, removing data from a slot includes: storing the data in the slot in an external memory and deleting the data in the slot.

[0201] In some embodiments, the KV cache manager determines the slot for evicted data based on at least one of the following strategies: least recently used (LRU) strategy and first-in first-out (FIFO) strategy.

[0202] In some embodiments, during step 520, the KV cache manager sets the attribute of the slot containing the key-value vector of at least one second initial lexical corresponding to each layer in the first layer set to pre-filled. After completing step 520, the KV cache manager updates the attribute of the slot containing the key-value vector of at least one second initial lexical corresponding to each layer in the first layer set from pre-filled to decoded. Similarly, during step 530, the KV cache manager sets the attribute of the slot containing the key-value vector of at least one second initial lexical corresponding to each layer in the third layer set to pre-filled. After completing step 530, the KV cache manager updates the attribute of the slot containing the key-value vector of at least one second initial lexical corresponding to each layer in the third layer set from pre-filled to decoded.

[0203] Figure 7 is a schematic structural diagram of a computing device provided in an embodiment of this application. The computing device 700 in Figure 7 includes a transceiver module 710 and a processing module 720. The computing device 700 in Figure 7 is used to run large models. The computing device 700 in Figure 7 can be used to execute the methods in Figure 3 or Figure 5. The computing device 700 in Figure 7 is, for example, the server in Figure 2.

[0204] When the computing device 700 executes the method in FIG3, the transceiver module 710 executes step 310, and the processing module 720 executes steps 320-330.

[0205] When the computing device 700 executes the method in FIG5, the transceiver module 710 executes step 510, and the processing module 720 executes steps 520-550.

[0206] In some embodiments, the computing device 700 further includes a storage management module 730. This storage management module 730 is used to execute the memory management methods described above in the large model inference process; that is, the storage management module 730 is used to execute the methods that the KV cache manager can execute.

[0207] The transceiver module 710, processing module 720, and storage management module 730 can all be implemented in software or in hardware. For example, the implementation of processing module 720 will be described below. Similarly, the implementation of transceiver module 710 and storage management module 730 can refer to the implementation of processing module 720.

[0208] As an example of a software functional unit, the processing module 720 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the processing module 720 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0209] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0210] As an example of a hardware functional unit, the processing module 720 may include at least one computing device, such as a server. Alternatively, the processing module 720 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0211] The processing module 720 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 720 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 720 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0212] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0213] It should be noted that the computing device provided in the above embodiments is only illustrated by the division of the functional modules described above when executing the above methods. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computing device can be divided into different functional modules to complete all or part of the functions described above. For example, the transceiver module 710 can be used to execute any step in the above methods, the processing module 720 can be used to execute any step in the above methods, and the storage management module 730 can be used to execute any step in the above methods. The steps implemented by the transceiver module 710, the processing module 720, and the storage management module 730 can be specified as needed. By implementing different steps in the above methods through the transceiver module 710, the processing module 720, and the storage management module 730, all the functions of the above computing device can be realized.

[0214] Furthermore, the computing device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0215] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0216] Figure 8 is a schematic structural block diagram of a computing device 800 provided in an embodiment of this application. The computing device 800 can be a server, a computer, or other device with computing capabilities. The computing device 800 shown in Figure 8 includes at least one processor 810 and a memory 820.

[0217] It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0218] The processor 810 executes instructions in the memory 820, causing the computing device 800 to implement the method provided in this application. Alternatively, the processor 810 executes instructions in the memory 820, causing the computing device 800 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0219] Optionally, the computing device 800 also includes a communication interface 830. The communication interface 830 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 800 and other devices or communication networks.

[0220] Optionally, the computing device 800 also includes a system bus 840, wherein the processor 810, memory 820, and communication interface 830 are respectively connected to the system bus 840. The processor 810 can access the memory 820 through the system bus 840; for example, the processor 810 can perform data read / write or code execution in the memory 820 through the system bus 840. The system bus 840 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 840 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in Figure 8, but this does not mean that there is only one bus or one type of bus.

[0221] In one possible implementation, the processor 810 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in the memory 820 or the cache of the processor 810.

[0222] Optionally, the processor 810 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, the processor 810 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Among these, a general-purpose processor is a microprocessor, etc. For example, the processor 810 may be a central processing unit (CPU).

[0223] The memory 820 provides runtime space for processes in the computing device 800. For example, the memory 820 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 820. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 820 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0224] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 810, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 810 will load the data required for processing into RAM for computation, and then transfer the result back out after the computation is complete.

[0225] By way of example and not limitation, memory 820 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile storage medium may be, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus DRAM (DRDRAM). It should be noted that the memory 820 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0226] The structures of the computing device 800 listed above are merely illustrative and are not limited thereto. The computing device 800 in this application includes various hardware components in existing computer systems. For example, the computing device 800 also includes other memories besides the memory 820, such as disk storage. Those skilled in the art should understand that the computing device 800 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 800 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computing device 800 may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in FIG8.

[0227] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0228] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 820 of one or more computing devices 800 in the computing device cluster may store the same instructions for performing the above-described methods.

[0229] In some possible implementations, the memory 820 of one or more computing devices 800 in the computing device cluster may also each store a portion of the instructions for executing the above-described method. In other words, a combination of one or more computing devices 800 can jointly execute the instructions of the above-described method.

[0230] It should be noted that the memories 820 in different computing devices 800 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned computing device. That is, the instructions stored in the memories 820 of different computing devices 800 can implement the functions of one or more modules within the aforementioned computing device.

[0231] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices, 800A and 800B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device.

[0232] It should be understood that the functions of computing device 800A shown in Figure 10 can also be performed by multiple computing devices 800. Similarly, the functions of computing device 800B can also be performed by multiple computing devices 800.

[0233] In this embodiment of the application, a computer program product containing instructions is also provided. The computer program product may be software or program products containing instructions capable of running on a computing device cluster or stored on any available medium. When run by the computing device cluster, it causes the computing device cluster to perform the methods provided above, or causes the computing device cluster to implement the functions of the computing device provided above.

[0234] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., high-density digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method provided above.

[0235] In this application embodiment, a chip system is also provided. The chip system includes logic circuitry for coupling with an input / output interface to transmit data via the input / output interface, thereby executing the methods provided above.

[0236] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0237] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0239] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0241] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0242] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

A method for large-scale model reasoning, characterized in that, The method is applied to large models, and the method includes: Receive a first request message, which includes H first initial tokens, where H is a positive integer; The first word set is decoded to determine the first target word. The first word set includes at least one first initial word and at least one target word from the H first initial word sets. The at least one target word is determined based on the at least one first initial word. The large model includes at least one layer, and the at least one layer includes a target layer. Based on the at least one target word and the first target word, determine the first response information; The step of decoding the first word set to determine the first target word includes: Based on the first set of lexical units, a first query vector and N key-value vector sets corresponding to the target layer are determined. The N key-value vector sets include the key-value vector of each first initial lexical unit in the at least one first initial lexical unit. Each key-value vector set in the N key-value vector sets includes at least one key-value vector. The key-value vector of each first initial lexical unit is determined based on the at least one first initial lexical unit. N is a positive integer. Based on the first query vector corresponding to the target layer and N key-value vector sets, a first attention score and a first output result corresponding to the first attention score are determined. The first attention score is determined based on the attention score corresponding to each of the M first key-value vector sets. The M first key-value vector sets belong to the N key-value vector sets, where M is a positive integer and M≤N. The first attention score is determined to be greater than or equal to a first preset threshold, and the first target word is determined based on the first output result corresponding to the first attention score. The method according to claim 1, characterized in that, The first attention score is determined by the sum of the attention scores of each of the M first key-value vector sets and a first value, whereby the first value represents the sum of the attention scores of each of the N key-value vector sets. The method according to claim 2, characterized in that, The first value is determined by the sum of the attention scores of each of the M first key value vector sets and a second value. The second value is determined by the minimum attention score among the M first key value vector sets and a third value, where the third value is the difference between N and M. The method according to any one of claims 1 to 3, characterized in that, The first importance of each of the M first key-value vector sets is higher than or equal to the first importance of the key-value vector sets other than the M first key-value vector sets in the N key-value vector sets. The first importance of the key-value vector set is determined based on the metadata vector of the key-value vector set and the first query vector. The metadata vector of the key-value vector set is determined based on the key-value vector set. The method according to any one of claims 1 to 4, characterized in that, The at least one target word includes a second target word, which is determined by pre-filling the at least one first initial word; or... The at least one target word includes a second target word and a third target word. The second target word is determined by pre-filling the at least one first initial word. The set of N key-value vectors also includes the key-value vector of the second target word. The key-value vector of the second target word and the third target word are determined by decoding the at least one first initial word and the second target word. The method according to any one of claims 1 to 5, characterized in that, At least one layer of the large model belongs to P sets of layers, and each of the P sets of layers includes one or more layers from the at least one layer, where P is a positive integer. The method further includes: Receive a second request message, which includes G second initial tokens, where G is a positive integer; The decoding process of the first word set to determine the first target word includes: Based on at least one of the G second initial lexical units and the first lexical set, a second output result and the first target lexical unit are obtained. The second output result is determined by pre-filling the at least one second initial lexical unit according to a first-layer set, where the first-layer set belongs to the P layer sets; or... Based on the third output result and the first word set, a fourth output result and the first target word are obtained. The third output result is the output of the second layer set. The fourth output result is determined by pre-filling the third output result based on the third layer set. The second layer set and the third layer set belong to different layer sets among the P layer sets. The method according to claim 6, characterized in that, The method further includes: The fourth target word is determined based on the second output result or the fourth output result; The second response information is determined based on the fourth target word. The method according to any one of claims 1 to 7, characterized in that, The large model includes multiple layers, and at least two key-value vectors corresponding to different layers share a first slot in a first storage space. The first storage space includes at least one slot, and each of the at least one slot is used to store at least one key-value vector. The first slot belongs to the at least one slot. A computing device, characterized in that, The computing device is used to run large models, and the computing device includes: The transceiver module is used to receive first request information, which includes H first initial tokens, where H is a positive integer; The processing module is used to decode the first word set to determine the first target word. The first word set includes at least one first initial word and at least one target word from the H first initial word sets. The at least one target word is determined based on the at least one first initial word. The large model includes at least one layer, and the at least one layer includes a target layer. The processing module is further configured to determine first response information based on the at least one target word and the first target word; Specifically, the processing module is used for: Based on the first set of lexical units, a first query vector and N key-value vector sets corresponding to the target layer are determined. The N key-value vector sets include the key-value vector of each first initial lexical unit in the at least one first initial lexical unit. Each key-value vector set in the N key-value vector sets includes at least one key-value vector. The key-value vector of each first initial lexical unit is determined based on the at least one first initial lexical unit. N is a positive integer. Based on the first query vector corresponding to the target layer and N key-value vector sets, a first attention score and a first output result corresponding to the first attention score are determined. The first attention score is determined based on the attention score corresponding to each of the M first key-value vector sets. The M first key-value vector sets belong to the N key-value vector sets, where M is a positive integer and M≤N. The first attention score is determined to be greater than or equal to a first preset threshold, and the first target word is determined based on the first output result corresponding to the first attention score. The computing device according to claim 9, characterized in that, The first attention score is determined by the sum of the attention scores of each of the M first key-value vector sets and a first value, whereby the first value represents the sum of the attention scores of each of the N key-value vector sets. The computing device according to claim 10, characterized in that, The first value is determined by the sum of the attention scores of each of the M first key value vector sets and a second value. The second value is determined by the minimum attention score among the M first key value vector sets and a third value, where the third value is the difference between N and M. The computing device according to any one of claims 9 to 11, characterized in that, The first importance of each of the M first key-value vector sets is higher than or equal to the first importance of the key-value vector sets other than the M first key-value vector sets in the N key-value vector sets. The first importance of the key-value vector set is determined based on the metadata vector of the key-value vector set and the first query vector. The metadata vector of the key-value vector set is determined based on the key-value vector set. The computing device according to any one of claims 9 to 12, characterized in that, The at least one target word includes a second target word, which is determined by pre-filling the at least one first initial word; or... The at least one target word includes a second target word and a third target word. The second target word is determined by pre-filling the at least one first initial word. The set of N key-value vectors also includes the key-value vector of the second target word. The key-value vector of the second target word and the third target word are determined by decoding the at least one first initial word and the second target word. The computing device according to any one of claims 9 to 13, characterized in that, At least one layer of the large model belongs to a set of P layers, and each of the P layer sets includes one or more layers of the at least one layer, where P is a positive integer. The transceiver module is also used to receive second request information, which includes G second initial tokens, where G is a positive integer. The processing module is specifically used for: Based on at least one of the G second initial lexical units and the first lexical set, a second output result and the first target lexical unit are obtained. The second output result is determined by pre-filling the at least one second initial lexical unit according to the first layer set. The first layer set belongs to the P layer sets. or, Based on the third output result and the first word set, a fourth output result and the first target word are obtained. The third output result is the output of the second layer set. The fourth output result is determined by pre-filling the third output result based on the third layer set. The second layer set and the third layer set belong to different layer sets among the P layer sets. The computing device according to claim 14, characterized in that, The processing module is further configured to: The fourth target word is determined based on the second output result or the fourth output result; The second response information is determined based on the fourth target word. The computing device according to any one of claims 9 to 15, characterized in that, The large model includes multiple layers, and at least two key-value vectors corresponding to different layers share a first slot in a first storage space. The first storage space includes at least one slot, and each of the at least one slot is used to store at least one key-value vector. The first slot belongs to the at least one slot. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 8. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 8.

Images