Inference method and apparatus

Abstract

The present application relates to the field of computers, and discloses an inference method and apparatus, used for improving the throughput of an inference system and ensuring the high quality of inference results while reducing the first‑token inference latency. The method comprises: sending to a cloud server a prompt corresponding to input content; then receiving from the cloud server first X tokens obtained by performing inference on the prompt, and an intermediate result, wherein the intermediate result comprises one or more of the following: a first ID sequence of tokens inputted by the cloud server performing full reasoning to acquire the first token, and a KV cache of first Y tokens among the first X tokens obtained by the cloud server performing inference; then, outputting the first X tokens; and performing inference on the basis of the obtained intermediate result, so as to obtain the remaining tokens in an answer corresponding to the prompt and to output same.

Description

A reasoning method and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202510127605.5, filed with the State Intellectual Property Office of China on January 27, 2025, entitled "A Reasoning Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computers, and more particularly to a reasoning method and apparatus. Background Technology

[0003] Large language models (LLMs) are the core of various intelligent question-answering systems and intelligent assistant systems. Users or system programs can input questions into the LLM, and the LLM will derive the answer through reasoning and output it.

[0004] LLM consists of several Transformer layers. Each Transformer layer employs a self-attention mechanism, calculating sequentially and completing one inference step to output a token. This process is repeated multiple times to arrive at the answer to the current question. The LLM reasoning process is an autoregressive inference, meaning it infers the next character based on past characters. The input is converted into prompts, which are then input into the LLM. The process of obtaining the first character through one inference step can be called full inference. The output of this inference becomes the input for the next inference step, and this process of inferring the next character is called incremental inference.

[0005] Inference latency is a crucial performance characteristic of LLM. If the answer to a problem contains N characters, then the total inference latency is: full inference latency + (N-1) * incremental inference latency. Currently, both cloud servers and edge devices can implement LLM functionality, but their performance differs significantly. Cloud-based LLM is typically large-scale, characterized by high-quality inference results and low full inference latency; however, the full inference latency in the cloud increases with the size of the input content. Furthermore, if multiple statements are inferred simultaneously during LLM inference in the cloud, the inference latency increases proportionally. Therefore, cloud-based throughput and inference latency are mutually constrained. Edge-based LLM, limited by its size, suffers from high full inference latency, while its incremental inference latency is close to that of the cloud.

[0006] The industry aims to shorten the latency of the first character of inference, improve the throughput of the inference system, and ensure the high quality of the inference results. Summary of the Invention

[0007] This application provides a reasoning method and apparatus that adopts an end-to-cloud collaborative approach. The cloud-assisted end includes partial word units of the first character and intermediate results. While shortening the latency of the first character in reasoning, it improves the throughput of the reasoning system and ensures the high quality of the reasoning results.

[0008] Firstly, a reasoning method is provided, which can be applied to an end device or a reasoning application within the end device. The method may include: sending a prompt word corresponding to the input content to a cloud server; then, receiving the first X lexical units obtained by the cloud server through reasoning on the prompt word, and intermediate results from the cloud server. The intermediate results include one or more of the following: a first identity (ID) sequence of the lexical units used to obtain the first character of the full-scale reasoning input; and a key-value cache of the first Y lexical units from the first X lexical units obtained by cloud reasoning. Afterwards, the end device outputs the first X lexical units; and performs reasoning based on the obtained intermediate results to obtain and output the remaining lexical units in the answer corresponding to the prompt word. Where X is greater than or equal to 1; Y is greater than or equal to 1 and less than or equal to X. Full-scale reasoning is used to obtain the first character in the answer corresponding to the prompt word.

[0009] The solution provided in this application employs an edge-cloud collaborative approach for inference. The cloud server performs the inference of the first X characters, including the first character, thus reducing the latency of full inference. The cloud server feeds back intermediate results during the inference process. Due to the short latency of the first character inference on the cloud server, the latency of the first character inference is significantly reduced. Furthermore, because the cloud server is large-scale and has high inference accuracy, the edge device can perform subsequent inference based on the intermediate results fed back by the cloud server, thereby improving the quality of the edge device's inference results. Moreover, in scenarios with a large number of inference output characters, a large number of incremental inferences can be deployed on the edge device, greatly reducing the pressure on the cloud device and significantly improving its throughput. This effectively achieves both reduced latency of the first character inference and improved throughput of the inference system, while ensuring high-quality edge device inference results.

[0010] In this context, the KV cache serves as an intermediate result of the inference process. K (query key) is used to identify the target candidate configured during inference. The similarity between the input and the candidate output is obtained by calculating the input content and K, also known as the attention score. This score is then applied to V (output value).

[0011] One possible implementation involves using the intermediate result as a first ID sequence. Reasoning is then performed based on this intermediate result, including: full reasoning and subsequent incremental reasoning based on the first ID sequence. Each incremental reasoning step is used to obtain one word after the first character. In this implementation, the content on the cloud-assisted device consists only of the ID sequence. If the number of IDs in this ID sequence is less than the number of IDs in the ID sequence corresponding to the prompt word, the data processing scale during on-device reasoning can be reduced.

[0012] Another possible implementation is as follows: Y is 1, meaning the intermediate result includes only the KV cache from the full inference process performed by the cloud server. The number of layers in the LLM on the end device is Q. The intermediate result includes the KV cache from layer 1 to layer a during the full inference process performed by the cloud server. Inference is performed based on the intermediate result to obtain and output the remaining tokens in the answer corresponding to the above prompt word. This includes: determining the KV cache from layer 1 to layer a used by the end LLM for full inference based on the KV cache from layer 1 to layer a during the full inference process performed by the cloud server. The end device performs the Yth incremental inference and subsequent incremental inference based on the reference ID sequence and the KV cache from layer 1 to layer a used by the LLM for full inference, until the stopping condition is met. The inference results of the Xth and subsequent incremental inferences are used as the remaining tokens in the answer and output. Wherein, if the intermediate result includes the first ID sequence, the reference ID sequence is the first ID sequence; if the intermediate result does not include the first ID sequence, the reference ID sequence is the second ID sequence of the tokens corresponding to the above prompt word.

[0013] Another possible implementation is that Y is 1, meaning the intermediate results include only the KV cache used by the cloud server during the full inference process. The number of layers in the LLM on the end device is Q. The intermediate results include the KV cache from the first layer to the a-th layer during the full inference process on the cloud server, where a is Q. Based on the reference ID sequence and the KV cache from the first layer to the a-th layer used by the LLM to perform full inference, the Y-th incremental inference is performed, which includes: inputting the reference ID sequence into the LLM on the end side, and performing the Y-th incremental inference on the end side based on the KV cache from the first layer to the a-th layer used by the LLM on the end side to perform full inference. Since the intermediate results of the cloud-assisted terminal include the same number of layers of the KV cache used for full inference on the cloud side as the number of layers of the LLM on the terminal side, the terminal side only needs to restore the KV cache used for full inference on the cloud side to the KV cache used for full inference on its own side, and then perform the Yth incremental inference (i.e. the first incremental inference after full inference). The terminal side does not need to perform full inference, which improves the processing efficiency of the terminal side; while the cloud side does not need to perform or performs incremental inference only in small amounts, which improves the throughput of the cloud side.

[0014] Another possible implementation is where Y is 1, meaning the intermediate results include only the KV cache from the full inference process performed by the cloud server. The LLM in the end device has Q layers, and the intermediate results include the KV cache from layer 1 to layer a during the full inference process performed by the cloud server, where a is less than Q. Based on the reference ID sequence and the KV cache from layer 1 to layer a used by the LLM for full inference, the Yth incremental inference is performed, including: inputting the reference ID sequence into the LLM; based on the KV cache from layer 1 to layer a used by the end LLM for full inference, performing full inference from layer a+1 to layer Q on the end-side LLN; and obtaining the KV cache from layer a+1 to layer Q performed by the end-side LLM for full inference. Then, based on the reference ID sequence and the KV cache from layer 1 to layer Q performed by the end-side LLM for full inference, the end-side performs the Yth incremental inference. Because the intermediate results from the cloud-assisted endpoint include fewer layers of the KV cache used for full inference on the cloud side compared to the number of layers in the endpoint LLM (only layers 1 to 'a'), the endpoint can restore the KV cache used for full inference on the cloud side to the local KV cache used for full inference on layers 1 to 'a', and then continue executing the subsequent layers of full inference (layers 'a+1' to 'Q'). Afterward, the Yth incremental inference (i.e., the first incremental inference after full inference) can be performed. The endpoint does not need to perform complete full inference, improving its processing efficiency; and the cloud side does not need to perform incremental inference, improving its throughput.

[0015] Another possible implementation involves an LLM with Q layers in the terminal device and Y greater than 1. The intermediate results from the cloud-assisted terminal include the KV caches from layers 1 to Q during the full inference and the first Y-2 incremental inferences performed by the cloud server, as well as the KV caches from layers 1 to b during the Y-1 incremental inference performed by the cloud server. Here, b is equal to Q. Inference is then performed based on these intermediate results to obtain and output the remaining tokens in the answer corresponding to the aforementioned prompt words. This includes: determining the KV caches from layers 1 to Q during the full inference and the first Y-1 incremental inference performed by the cloud server; inputting the reference ID sequence into the terminal device's LLM; performing the Y-th incremental inference and subsequent incremental inferences based on the KV caches from layers 1 to Q during the full inference and the first Y-1 incremental inference performed by the terminal device's LLM; and outputting the inference results from the X-th and subsequent incremental inferences as the remaining tokens in the answer. Since the intermediate results of the cloud-assisted terminal include the same number of KV cache layers for the full inference and the first Y-1 incremental inference operations performed on the cloud side as the number of LLM layers on the terminal side, the terminal side only needs to restore the KV cache for the full inference and the first Y-1 incremental inference operations performed on the cloud side to the KV cache for the full inference and the first Y-1 incremental inference operations performed on the local side, and then perform the Yth incremental inference operation. The terminal side does not need to perform full inference and the first Y-1 incremental inference operations, which improves the processing efficiency of the terminal side; and the cloud side does not need to perform a large number of incremental inference operations, which improves the throughput of the cloud side.

[0016] Another possible implementation involves an LLM with Q layers in the endpoint device, where Y is greater than 1. The intermediate results from the cloud-assisted endpoint include the KV caches from layers 1 to Q during the full inference and the first Y-2 incremental inference processes performed by the cloud server, as well as the KV caches from layers 1 to b during the Y-1 incremental inference process performed by the cloud server. b is less than Q. Inference is then performed based on these intermediate results to obtain and output the remaining tokens in the answer corresponding to the aforementioned prompt words. This includes: the endpoint device determining the KV caches from layers 1 to Q used in the full inference and the first Y-2 incremental inference processes performed by the cloud server, based on the KV caches from layers 1 to Q; and the endpoint LLM determining the KV caches from layers 1 to b used in the Y-1 incremental inference process performed by the cloud server. Next, based on the reference ID sequence and the KV caches of layers 1 to Q used by the endpoint LLM for full inference and the first Y-2 incremental inferences, as well as the KV caches of layers 1 to b during the Y-1 incremental inference, the endpoint LLM performs inference from layers b+1 to Q during the Y-1 incremental inference, and obtains the KV caches of layers b+1 to Q used by the endpoint LLM for the Y-1 incremental inference. Then, based on the reference ID sequence and the KV caches of layers 1 to Q used in the endpoint LLM for full inference and the first Y-1 incremental inferences, the endpoint LLM performs the Y-th incremental inference and subsequent incremental inferences until the stopping condition is met. The inference results of the Xth and subsequent incremental inferences are used as the remaining tokens in the above answer and output.

[0017] Since the intermediate results of the cloud-assisted terminal include the same number of KV cache layers for the full inference and the first Y-2 incremental inference operations on the cloud side as the terminal-side LLM, but the intermediate results include fewer KV cache layers for the Y-1 incremental inference operation on the cloud side (only including layers 1 to b) than the terminal-side LLM, the terminal can restore the KV cache for the full inference and the first Y-2 incremental inference operations on the cloud side to the KV cache for the full inference and the first Y-2 incremental inference operations on its own side; then restore the KV cache for layers 1 to b for the Y-1 incremental inference operation on the cloud side to the KV cache for the Y-1 incremental inference operation on the terminal-side LLM, and then continue to execute the subsequent layers of the Y-1 incremental inference (layers b+1 to Q). After that, the Y-th incremental inference can be executed. The edge does not need to perform full inference and the first Y-2 incremental inferences, but only needs to perform a portion of the layers in the Y-1 incremental inference, which improves the processing efficiency of the edge; while the cloud does not need to perform a large number of incremental inferences, which improves the throughput of the cloud.

[0018] Another possible implementation involves performing a first inference on the cloud server. This first inference can be either full or incremental. Based on the KV cache of the i-th layer of the first inference performed on the cloud server, the KV cache of the i-th layer of the LLM inference is determined. This includes: if the total number of parameters in the KV cache of the i-th layer of the first inference performed on the cloud server in the intermediate result is the same as the number of parameter positions in the KV cache of the i-th layer of the LLM inference, then the KV cache of the i-th layer of the first inference performed on the cloud server is used as the KV cache of the i-th layer of the LLM inference. Here, i is any value from 1 to Q. In this implementation, the parameters in the KV cache of the i-th layer of the first inference performed on the cloud server, including the intermediate result, can satisfy the number of positions in the KV cache of the i-th layer of the first inference performed on the client-side LLM. The client-side can then directly use the intermediate result as the KV cache of the i-th layer of the first inference performed on the client-side LLM without additional processing, resulting in high recovery efficiency and accuracy.

[0019] Another possible implementation involves performing a first inference on the cloud server. This first inference can be either full or incremental. Based on the KV cache of the i-th layer of the first inference performed on the cloud server, the KV cache of the i-th layer of the LLM inference is determined. This includes: if the total number of parameters in the KV cache of the i-th layer of the first inference performed on the cloud server in the intermediate result is less than the number of parameter positions in the KV cache of the i-th layer of the LLM inference, the KV cache of the i-th layer of the first inference performed on the cloud server is padded and used as the KV cache of the i-th layer of the LLM inference. In this implementation, if the parameters in the KV cache of the i-th layer of the first inference performed on the cloud-assisted terminal do not meet the number of positions in the KV cache of the i-th layer of the first inference performed on the terminal, the terminal can padded the parameters in the KV cache of the i-th layer of the first inference performed on the intermediate result and use it as the KV cache of the i-th layer of the first inference performed on the terminal. This allows the purpose of the cloud-assisted terminal intermediate result to be achieved by transmitting a small amount of data, thus reducing the amount of data transmitted and improving the inference quality on the terminal.

[0020] Another possible implementation, the above filling includes: filling multiple parameter positions in the KV cache of the i-th layer of the cloud server performing the first inference with a value.

[0021] Another possible implementation, the reasoning method provided in this application, may further include: obtaining input content; converting the input content into candidate prompt words; and preprocessing the candidate prompt words to obtain the aforementioned prompt words. The preprocessing may include one or more of the following operations: desensitization, compression, or encryption. The desensitization operation is used to replace sensitive words with corresponding characters in the desensitization rule. The desensitization rule includes one or more sensitive words and their corresponding characters.

[0022] Another possible implementation, the reasoning method provided in this application, may further include: sending transmission characteristics and / or specification information to a cloud server. Transmission characteristics are used to indicate the amount of data that can be transmitted between the end device and the cloud server. Specification information is used to indicate the cloud-side model specifications corresponding to the characteristics of the input content.

[0023] Another possible implementation involves transmission characteristics including the network status between the end device and the cloud server, and / or the user level corresponding to the end device.

[0024] Secondly, another reasoning method is provided, which can be applied to cloud servers. This method includes: receiving a prompt word from an end device; performing full reasoning and the first X-1 incremental reasoning iterations on the prompt word to obtain the first X lexical units; sending the first X lexical units and intermediate results to the end device, so that the end device can perform reasoning based on the intermediate results. The intermediate results include one or more of the following: the first ID sequence of the lexical units input to the full reasoning of the cloud server; and the KV cache of the first Y lexical units among the first X lexical units obtained by the cloud server reasoning. X is greater than or equal to 1. Y is greater than or equal to 1 and less than or equal to X. Full reasoning is used to obtain the first character of the answer corresponding to the above prompt word.

[0025] The solution provided in this application employs an edge-cloud collaborative approach for inference. The cloud server performs inference of the first X characters, including the first character, thus reducing the latency of full inference. The cloud server feeds back intermediate results during the inference process. Due to the large scale of the cloud server and the high accuracy of the inference results, the edge device executes subsequent inference based on the intermediate results fed back by the cloud server, which can improve the quality of the edge device's inference results. Furthermore, in scenarios with a large number of inference output characters, a large number of incremental inferences can be deployed on the edge device, greatly reducing the pressure on the cloud device and significantly improving its throughput. This effectively achieves both reduced latency for the first character inference and improved throughput of the inference system, while ensuring high-quality inference results.

[0026] One possible implementation of the inference method provided in this application may further include: determining the target data volume corresponding to the transmission characteristics in the first rule based on the transmission characteristics between the cloud server and the terminal device; and determining intermediate results whose data volume is less than the target data volume. The transmission characteristics are used to indicate the amount of data that can be transmitted between the terminal device and the cloud server. The size of the intermediate results from the cloud-assisted terminal is determined by the transmission characteristics between the cloud and the terminal, so that the intermediate results from the cloud-assisted terminal can be quickly transmitted to the terminal side, improving inference efficiency and reducing the initial latency and overall latency of inference.

[0027] In another possible implementation, the LLM in the end device is layer Q, Y is 1. The intermediate results where the data volume is less than the target data volume are determined as follows: In the LLM of the cloud server, the first i consecutive layers are identified as sparseable layers; the number of parameters in the j-th layer of the first i layers of the LLM in the cloud server is greater than or equal to the number of parameters in the j-th layer of the first i layers of the LLM in the end device, where j iterates from 1 to i; the number of parameters in the (i+1)-th layer of the LLM in the cloud server is less than the number of parameters in the i-th layer of the LLM in the end device, or i equals Q. The number of parameters in the KV cache of each sparseable layer performing full inference is reduced to obtain the target KV cache for each sparseable layer performing full inference. The number of parameters in the target KV cache of a sparseable layer performing full inference is equal to the number of parameters in the same position layer of the LLM in the end device; from the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparseable layer performing full inference, intermediate results where the data volume is less than or equal to the target data volume are determined.

[0028] In another possible implementation, the intermediate result of determining that the data volume is less than or equal to the target data volume from the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparse layer for full inference can include: if the data volume of the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparse layer for full inference is less than or equal to the target data volume, the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparse layer for full inference are determined as intermediate results; the first ID sequence is the lexical ID sequence corresponding to the prompt word.

[0029] In another possible implementation, determining an intermediate result where the data size is less than or equal to the target data size from the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparser layer performing full inference can include: if the data size of the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparser layer performing full inference is greater than the target data size, performing a sparsification operation on the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparser layer performing full inference to obtain an intermediate result where the data size is less than or equal to the target data size. The sparsification operation includes one or more of the following operations: deleting one or more layers of KV cache, or sparsifying one or more layers of KV cache.

[0030] In another possible implementation, the LLM in the end device is a Q layer, Y is greater than 1. The intermediate result of determining that the data volume is less than the target data volume includes: determining that the first i consecutive layers in the LLM of the cloud server are sparseable layers. The number of parameters in the KV cache of each sparseable layer performing full inference and the first Y-1 incremental inferences is reduced to obtain the target KV cache for each of the sparseable layers performing full inference and the first Y-1 incremental inferences. The number of parameters in the target KV cache of a sparseable layer performing full inference and the first Y-1 incremental inferences is equal to the number of parameters in the same position layer of the LLM in the end device. The intermediate result of determining that the data volume is less than or equal to the target data volume is derived from the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparseable layer performing full inference and the first Y-1 incremental inferences.

[0031] In another possible implementation, determining the intermediate result whose data volume is less than or equal to the target data volume from the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparser layer after performing full inference and the first Y-1 incremental inferences includes: if the lexical ID sequence corresponding to the prompt word and the data volume of the target KV cache of the sparser layer after performing full inference and the first Y-1 incremental inferences are less than or equal to the target data volume, then the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparser layer after performing full inference and the first Y-1 incremental inferences are determined as intermediate results. The first ID sequence is the lexical ID sequence corresponding to the prompt word.

[0032] In another possible implementation, determining an intermediate result where the data volume is less than or equal to the target data volume from the lexical ID sequence corresponding to the prompt word, the target KV cache of the sparse layer after performing full inference and the first Y-1 incremental inferences, includes: if the data volume of the lexical ID sequence corresponding to the prompt word, the target KV cache of the sparse layer after performing full inference and the first Y-1 incremental inferences is greater than the target data volume, performing content sparsification on the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparse layer of the first Y-2 incremental inferences, or performing a sparsification operation on the target KV cache of the sparse layer of the Y-1th incremental inference, to obtain an intermediate result where the data volume is less than or equal to the target data volume.

[0033] Another possible implementation, the reasoning method provided in this application may also include: transmission features sent by the receiving device.

[0034] Another possible implementation, the inference method provided in this application, may further include: specification information sent by the receiving device, which indicates the cloud-side model specification corresponding to the features of the input content. Accordingly, performing full inference and the first X-1 incremental inferences on the prompt words includes: performing full inference and the first X-1 incremental inferences on the prompt words according to the model specification indicated by the specification information. The specifications of the cloud-side LLM can differ from those of the edge-side LLM, using a larger-specification, higher-quality model to assist the edge and achieve better inference results.

[0035] Another possible implementation involves receiving a prompt word from an end device, including: receiving a preprocessed prompt word from the end device. Preprocessing may include compression or encryption. Correspondingly, performing full inference and the first X-1 incremental inferences on the prompt word includes: performing the inverse operation of the preprocessing on the prompt word, and then performing full inference and the first X-1 incremental inferences again.

[0036] Thirdly, an inference apparatus is provided, which can be applied to an end device or an inference application within an end device. The apparatus may include: a sending unit, an acquiring unit, an output unit, and an inference unit. Wherein:

[0037] The sending unit is used to send prompt words corresponding to the input content to the cloud server.

[0038] The receiving unit receives the first X lexical units obtained by the cloud server through inference on the prompt words, as well as intermediate results, sent by the cloud server. The intermediate results include one or more of the following: the first ID sequence of the lexical units used to obtain the first character from the full inference input; and a KV cache of the first Y lexical units from the first X lexical units obtained by the cloud server. Where X is greater than or equal to 1; Y is greater than or equal to 1 and less than or equal to X. The full inference is used to obtain the first character in the answer corresponding to the prompt words.

[0039] The output unit is used to output the first X words.

[0040] The reasoning unit is used to perform reasoning based on the obtained intermediate results to obtain and output the remaining word units in the answer corresponding to the above prompt words.

[0041] One possible implementation involves using the first ID sequence as the intermediate result. The inference unit is specifically used to: perform full inference and subsequent incremental inference based on the first ID sequence. Each incremental inference step is used to obtain one word after the first character.

[0042] Another possible implementation is that Y is 1, meaning the intermediate result includes only the KV cache from the cloud server's full inference process. The LLM layer in the end device is Q, and the intermediate result includes the KV cache from layer 1 to layer a from the cloud server's full inference process. Specifically, the inference unit is used to: determine the KV cache from layer 1 to layer a used by the end LLM for full inference based on the KV cache from layer 1 to layer a in the cloud server's full inference process. The end device performs the Yth incremental inference and subsequent incremental inferences based on the reference ID sequence and the KV cache from layer 1 to layer a used by the LLM for full inference, until the stopping condition is met. The inference results of the Xth and subsequent incremental inferences are used as the remaining tokens in the answer and output. Wherein, if the intermediate result includes the first ID sequence, the reference ID sequence is the first ID sequence; if the intermediate result does not include the first ID sequence, the reference ID sequence is the second ID sequence corresponding to the aforementioned prompt word.

[0043] Another possible implementation is that Y is 1, meaning the intermediate results include only the KV cache used by the cloud server during the full inference process. The number of layers in the LLM on the end device is Q. The intermediate results include the KV cache from the first layer to the a-th layer during the full inference process on the cloud server, where a is Q. Based on the reference ID sequence and the KV cache from the first layer to the a-th layer used by the LLM to perform full inference, the Y-th incremental inference is performed, which includes: inputting the reference ID sequence into the LLM on the end side, and performing the Y-th incremental inference on the end side based on the KV cache from the first layer to the a-th layer used by the LLM on the end side to perform full inference.

[0044] Another possible implementation is where Y is 1, meaning the intermediate results include only the KV cache from the full inference process performed by the cloud server. The LLM in the end device has Q layers, and the intermediate results include the KV cache from layer 1 to layer a during the full inference process performed by the cloud server, where a is less than Q. Specifically, the aforementioned inference unit is used to: determine the KV cache from layer 1 to layer a used by the LLM for full inference based on the KV cache from layer 1 to layer a during the full inference process performed by the cloud server. If a is less than Q, the reference ID sequence is input into the LLM. Based on the KV cache from layer 1 to layer a used by the end LLM for full inference, the end-side LLN performs inference from layer a+1 to layer Q during full inference, obtaining the KV cache from layer a+1 to layer Q during full inference performed by the end-side LLM. Then, based on the reference ID sequence and the KV cache from layer 1 to layer Q during full inference performed by the end-side LLM, the Yth incremental inference is performed on the end-side device. Because the intermediate results from the cloud-assisted endpoint include fewer layers of the KV cache used for full inference on the cloud side compared to the number of layers in the endpoint's LLM (only including layers 1 to 'a'), the endpoint can restore the KV cache used for full inference on the cloud side to the local KV cache used for full inference on layers 1 to 'a', and then continue executing the subsequent layers of full inference (layers 'a+1' to 'Q'). Afterward, the Yth incremental inference (i.e., the first incremental inference after full inference) can be executed.

[0045] Another possible implementation involves an LLM with Q layers in the terminal device and Y greater than 1. The intermediate results from the cloud-assisted terminal include the KV caches from layers 1 to Q during the full inference and the first Y-2 incremental inference processes performed by the cloud server, as well as the KV caches from layers 1 to b during the Y-1 incremental inference process performed by the cloud server. Here, b is Q. Specifically, the inference unit is used to: if b is Q, determine the KV caches from layers 1 to Q during the full inference and the first Y-1 incremental inference processes performed by the cloud server; input the reference ID sequence into the terminal device's LLM, perform the Y-th incremental inference and subsequent incremental inference based on the KV caches from layers 1 to Q during the full inference and the first Y-1 incremental inference processes performed by the terminal device's LLM, and output the remaining terms in the answer.

[0046] Another possible implementation is that the number of layers in the LLM on the end device is Q, Y is greater than 1, and the intermediate results of the cloud-assisted end device include the KV cache of layers 1 to Q during the full inference and the first Y-2 incremental inference processes performed by the cloud server, as well as the KV cache of layers 1 to b during the Y-1 incremental inference process performed by the cloud server. b is less than Q. The above inference unit is specifically used for: the end device determining the KV cache of layers 1 to Q used for full inference and the first Y-2 incremental inference processes performed by the cloud server based on the KV cache of layers 1 to Q used for full inference and the first Y-2 incremental inference processes performed by the cloud server; and determining the KV cache of layers 1 to b used for the Y-1 incremental inference process performed by the end device based on the KV cache of layers 1 to b used for the Y-1 incremental inference process performed by the cloud server. Subsequently, based on the reference ID sequence and the KV caches of layers 1 to Q used by the edge LLM for performing full inference and the first Y-2 incremental inferences, as well as the KV caches of layers 1 to b during the Y-1 incremental inference, layer b+1 to Q inference is performed on the edge for the Y-1 incremental inference, and the KV caches of layers b+1 to Q during the Y-1 incremental inference are obtained from the edge LLM. Then, based on the reference ID sequence and the KV caches of layers 1 to Q from the first Y-1 incremental inferences performed by the edge LLM, the Y-th incremental inference is performed on the edge.

[0047] Another possible implementation involves performing a first inference on the cloud server. This first inference can be either a full inference or an incremental inference. Based on the KV cache of the i-th layer of the first inference performed on the cloud server, the KV cache of the i-th layer of the LLM inference is determined. This includes: if the total number of parameters in the KV cache of the i-th layer of the first inference performed on the cloud server is the same as the number of parameter positions in the KV cache of the i-th layer of the LLM inference, then the KV cache of the i-th layer of the first inference performed on the cloud server is used as the KV cache of the i-th layer of the LLM inference. Here, i is any value from 1 to Q.

[0048] Another possible implementation involves performing a first inference on the cloud server, which can be either a full inference or an incremental inference. Based on the KV cache of the i-th layer of the first inference performed on the cloud server, the KV cache of the i-th layer of the LLM inference is determined, including: if the total number of parameters in the KV cache of the i-th layer of the first inference performed on the cloud server is less than the number of parameter positions in the KV cache of the i-th layer of the LLM inference, the KV cache of the i-th layer of the first inference performed on the cloud server is filled and used as the KV cache of the i-th layer of the LLM inference.

[0049] Another possible implementation, the above filling includes: filling multiple parameter positions in the KV cache of the i-th layer of the cloud server performing the first inference with a value.

[0050] In another possible implementation, the aforementioned acquisition unit can also be used to: acquire input content; convert the input content into candidate prompt words; and preprocess the candidate prompt words to obtain the aforementioned prompt words. This preprocessing may include one or more of the following operations: desensitization, compression, or encryption. The desensitization operation is used to replace sensitive words with corresponding characters in the desensitization rules. The desensitization rules include one or more sensitive words and their corresponding characters.

[0051] In another possible implementation, the aforementioned sending unit is also used to: send transmission characteristics and / or specification information to the cloud server. The transmission characteristics indicate the amount of data that can be transmitted between the end device and the cloud server. The specification information indicates the cloud-side model specification corresponding to the characteristics of the input content.

[0052] Another possible implementation involves transmission characteristics including the network status between the end device and the cloud server, and / or the user level corresponding to the end device.

[0053] Fourthly, another inference device is provided, which can be applied to cloud servers. This device includes: a receiving unit, an inference unit, and a transmitting unit. Wherein:

[0054] The receiving unit is used to receive prompts from the end device.

[0055] The reasoning unit performs full reasoning and the first X-1 incremental reasoning iterations on the prompt words to obtain the first X lexical units. Full reasoning is used to obtain the first character of the answer corresponding to the above prompt words.

[0056] The sending unit is used to send the first X tokens and intermediate results to the end device, so that the end device can perform inference based on the intermediate results. The intermediate results include one or more of the following: the first ID sequence of the tokens input to the full inference by the cloud server; and the KV cache of the first Y tokens out of the first X tokens obtained by the cloud server inference. X is greater than or equal to 1. Y is greater than or equal to 1 and less than or equal to X.

[0057] In one possible implementation, the apparatus further includes a determining unit, configured to determine, based on the transmission characteristics between the cloud server and the end device, the target data volume corresponding to the transmission characteristics in the first rule; and to determine an intermediate result where the data volume is less than the target data volume. The transmission characteristics are used to indicate the supported data volume between the end device and the cloud server.

[0058] In another possible implementation, the LLM in the end device is layer Q, Y is 1, and the determining unit is specifically used to: determine the first i consecutive layers in the LLM of the cloud server as sparseable layers; the number of parameters in the j-th layer of the first i layers of the LLM in the cloud server is greater than or equal to the number of parameters in the j-th layer of the first i layers of the LLM in the end device, where j traverses from 1 to i; the number of parameters in the (i+1)-th layer of the LLM in the cloud server is less than the number of parameters in the i-th layer of the LLM in the end device, or i equals Q. The number of parameters in the KV cache of each sparseable layer performing full inference is reduced to obtain the target KV cache of each sparseable layer performing full inference, and the number of parameters in the target KV cache of a sparseable layer performing full inference is equal to the number of parameters in the same position layer of the LLM in the end device; from the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparseable layer performing full inference, intermediate results with a data volume less than the target data volume are determined.

[0059] In another possible implementation, the determining unit is specifically used to: if the word ID sequence corresponding to the prompt word and the data volume of the target KV cache of the sparse layer for performing full inference are less than or equal to the target data volume, determine the word ID sequence corresponding to the prompt word and the target KV cache of the sparse layer for performing full inference as intermediate results; the first ID sequence is the word ID sequence corresponding to the prompt word.

[0060] In another possible implementation, the determining unit is specifically used to: if the lexical ID sequence corresponding to the prompt word and the data volume of the target KV cache of the sparser layer performing full inference are greater than the target data volume, perform a sparsification operation on the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparser layer performing full inference, to obtain an intermediate result with a data volume less than or equal to the target data volume. The sparsification operation includes one or more of the following operations: deleting one or more KV cache layers, or sparsifying the KV cache in one or more layers.

[0061] In another possible implementation, the LLM in the end device is a Q layer, Y is greater than 1, and the determining unit is specifically used to: determine the first i consecutive layers in the LLM of the cloud server as sparseable layers. The number of parameters in the KV cache of each sparseable layer performing full inference and the first Y-1 incremental inferences is reduced to obtain the target KV cache of each sparseable layer performing full inference and the first Y-1 incremental inferences. The number of parameters in the target KV cache of a sparseable layer performing full inference and the first Y-1 incremental inferences is equal to the number of parameters in the same position layer of the LLM in the end device. From the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparseable layer performing full inference and the first Y-1 incremental inferences, intermediate results with a data volume less than the target data volume are determined.

[0062] In another possible implementation, the determining unit is specifically used to: if the token ID sequence corresponding to the prompt word and the data volume of the target KV cache of the sparse layer during full inference and the first Y-1 incremental inferences are less than or equal to the target data volume, determine the token ID sequence corresponding to the prompt word and the target KV cache of the sparse layer during full inference and the first Y-1 incremental inferences as intermediate results. The first ID sequence is the token ID sequence corresponding to the prompt word.

[0063] In another possible implementation, the LLM in the end device is a Q layer, Y is greater than 1, and the determining unit is specifically used to: determine the first i consecutive layers in the LLM of the cloud server as sparseable layers. The number of parameters in the KV cache of each sparseable layer performing full inference and the first Y-1 incremental inferences is reduced to obtain the target KV cache of each sparseable layer performing full inference and the first Y-1 incremental inferences. The number of parameters in the target KV cache of a sparseable layer performing full inference and the first Y-1 incremental inferences is equal to the number of parameters in the same position layer of the LLM in the end device. If the lexical ID sequence corresponding to the prompt word and the data volume of the target KV cache of the sparseable layer performing full inference and the first Y-1 incremental inferences are greater than the target data volume, content sparsification is performed on the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparseable layer performing the first Y-2 incremental inferences, or sparsification is performed on the target KV cache of the sparseable layer performing the Y-1 incremental inferences, resulting in an intermediate result where the data volume is less than or equal to the target data volume.

[0064] In another possible implementation, the receiving unit described above is also used for: transmission features sent by the receiving device.

[0065] In another possible implementation, the receiving unit is further configured to: receive specification information sent by the receiving device, which indicates the cloud-side model specification corresponding to the features of the input content. Correspondingly, the inference unit is specifically configured to: perform full inference and the first X-1 incremental inferences on the prompt words according to the model specification indicated by the specification information.

[0066] In another possible implementation, the receiving unit is specifically configured to receive a preprocessed prompt word from the end device. Preprocessing includes compression or encryption. Correspondingly, the inference unit is specifically configured to perform the inverse operation of the preprocessing on the prompt word, followed by full inference and the first X-1 incremental inferences.

[0067] Fifthly, an end device is provided, comprising a processor and a memory. The processor is configured to execute instructions stored in the memory to cause the end device to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0068] Sixthly, a cloud server is provided, comprising a processor and a memory. The processor executes instructions stored in the memory to cause the cloud server to perform operational steps as described in the second aspect or any possible implementation thereof.

[0069] In a seventh aspect, an inference system is provided, comprising an end device as described in the fifth aspect and a cloud server as described in the sixth aspect.

[0070] Eighthly, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a processor, causing the processor to perform operational steps of the method as described in the first aspect, the second aspect, or any of the possible implementations above.

[0071] Ninthly, a computer program product is provided that, when run on a computer, causes the computer to perform the operational steps of the method as described in the first aspect, the second aspect, or any of the possible implementations above.

[0072] The technical effects of any of the design methods in aspects three through nine can be found in the technical effects of different design methods in aspects one or two, and will not be repeated here.

[0073] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0074] Figure 1 is a schematic diagram of the application scenario of the solution provided in the embodiments of this application;

[0075] Figure 2a is a schematic diagram of the structure of a reasoning system provided in an embodiment of this application;

[0076] Figure 2b is a schematic diagram of another inference system provided in an embodiment of this application;

[0077] Figure 3 is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0078] Figure 4 is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0079] Figure 5 is a flowchart illustrating a reasoning method provided in an embodiment of this application;

[0080] Figure 6 is a schematic diagram of the process of determining a sparsifiable layer in a cloud server according to an embodiment of this application;

[0081] Figure 7 is a schematic diagram of a sparse intermediate result scheme provided in an embodiment of this application;

[0082] Figure 8 is a schematic diagram of a cloud-assisted terminal scenario provided in an embodiment of this application;

[0083] Figure 9 is a schematic diagram of the structure of a reasoning device provided in an embodiment of this application;

[0084] Figure 10 is a schematic diagram of another inference device provided in an embodiment of this application. Detailed Implementation

[0085] To facilitate understanding, the main terms used in this application will be explained first.

[0086] Large Language Models (LLMs): Models that can understand and generate natural language text through deep learning and large amounts of training data.

[0087] Prompt: LLM input with a specific format.

[0088] KV Cache (KV Cache) refers to the intermediate K / V results generated by LLM inference (key-value pairs cached in LLM).

[0089] Layers, also known as attention layers (or direct layers), are the core model structure that makes up LLM.

[0090] A token, also known as a character or word, is the basic unit of text in Natural Language Processing (NLP). For example, a token can be a word, punctuation mark, or phrase. Tokenization is the process of dividing text into these "tokens," providing a foundation for subsequent processing such as text analysis and machine translation. During LLM inference, tokens are represented internally by IDs.

[0091] Full inference: refers to the process from inputting the prompt word to deriving the first word in LLM inference. The result of full inference is called the prefill.

[0092] Incremental reasoning refers to the process in LLM reasoning where, after full reasoning, the next word is derived from the previous word as input.

[0093] LLM consists of several layers (Transformer layers), with the Attention mechanism at its core. During the inference process of a Transformer layer, the input is mapped to three tensors, Q, K, and V, through the Attention mechanism. Q represents the input, K represents the target candidate (query key), and the similarity between the input and the candidate output is calculated using Q and K, also known as the Attention Score. This score is then applied to V (the output value) as the output of that layer. An LLM completes one round of inference by executing the inference of each layer in sequence, completing one round of inference (a Transformer layer can contain independent Attention calculations across multiple dimensions, such as processing 32 Attentions simultaneously), and obtaining the result of one inference. For a given problem, the complete inference process of LLM includes one full inference and multiple incremental inferences. Both full and incremental inference can be viewed as a process of deriving the next character based on previous characters (autoregressive inference). LLM outputs the answer by continuously "jumping" characters, outputting the results of multiple inferences in the order of inference; this method is known in the industry as streaming.

[0094] Currently, both cloud servers and end devices (hereinafter referred to as cloud and end) can execute LLM, but they exhibit completely different performance characteristics.

[0095] The latency of full inference in the cloud increases with the size of the input text. Generally, for text input of less than 10,000 characters, the latency of full inference in the cloud is at most 1 second. Subsequent incremental inferences typically have a latency of tens of milliseconds per increment. Furthermore, when performing LLM inference, multiple sentences are processed simultaneously, thus increasing the inference latency proportionally. For example, with two 10,000-character questions input simultaneously, the latency of full inference in the cloud LLM is 1-2 seconds, while the latency of a single round of incremental inference, although also increasing, remains within tens of milliseconds. Due to this significant difference in inference latency between full and incremental inference, the cloud tends to: 1) use a maximum of 2-4 questions as input for full inference; otherwise, the latency from question input to the first character output is too large, resulting in a poor user experience; 2) use dozens or even hundreds of questions as input for incremental inference, with each character still taking tens to hundreds of milliseconds, meaning one character is output every hundred milliseconds, satisfying the user's reading needs. If the number of words in a question (output word count) is N, then the total inference latency is: full inference latency + (N-1) * incremental inference latency. Although cloud computing has strong processing capabilities, when the number of output words N is large, inferring a single question can consume tens of seconds of cloud resources. Cloud-based LLMs are generally large in size (model size is measured by the number of parameters, B (Billion) represents one billion parameters), typically tens or even hundreds of B. Currently, in order to simultaneously pursue low latency for full inference and high throughput for incremental inference, the mainstream solution is to separate full inference and incremental inference, using dedicated container instances to infer full or incremental inference separately (full and incremental inference are within the same container instance, and the number of questions inferred simultaneously between full and incremental inference is the same, making it impossible to simultaneously pursue low latency for full inference and high throughput for incremental inference).

[0096] The full inference latency on the client side is extremely high, approaching 10 seconds for problems with text larger than 4K. However, the incremental inference latency on the client side is still only tens of milliseconds. High latency in the first character leads to a poor user experience. The client-side LLM specification is relatively small, generally only a few bytes of parameters.

[0097] Based on the performance characteristics of cloud and edge inference execution, the industry typically adopts an edge-cloud collaborative approach to improve inference performance.

[0098] One edge-cloud collaborative inference scheme involves sending requests, such as long texts, which are slow to process on the end device, directly to the cloud for inference. The inference process in the cloud follows Private Cloud Computing (PCC), ensuring that user privacy information is not recorded. Resources are provided on demand in the cloud for inference requests, and all resources are released after inference is complete. PCC then returns the inference result to the end device. While this scheme protects user privacy during edge-cloud collaboration, its inference latency is positively correlated with the length of the complete text in the request, resulting in a relatively long latency from question input to complete answer output, making it unsuitable for latency-sensitive tasks. Furthermore, since the entire inference process is executed by the cloud, although both full and incremental inference latency are relatively short, the overall latency of a single request consuming cloud resources is long when the number of output characters is large. As the user base increases, cloud throughput becomes a bottleneck.

[0099] Another edge-cloud collaborative inference scheme involves the following steps: the edge infers several token texts; the edge's output tokens are uploaded to the cloud; the cloud performs batch scoring and selects the correct parts; the cloud sends the correct tokens back to the edge as the mark, and this process is repeated. This scheme involves frequent edge-cloud interactions, with both the edge and cloud needing to execute the complete inference process. The lack of acceleration for the first character on the edge results in significant latency. These frequent edge-cloud interactions greatly reduce inference efficiency and also lead to low cloud throughput.

[0100] Based on this, this application provides an inference method that performs inference through edge-cloud collaboration. The cloud server performs inference of the first X characters, including the first character, and feeds back the first X tokens to the end device, along with a key-value cache of the first Y tokens from the first X tokens obtained from the cloud server inference. In this way, the end device can utilize the cloud-assisted content to perform inference. Because the cloud server's inference of the first character has a short latency, the initial inference latency is significantly reduced. Furthermore, due to the large scale of the cloud server and the high accuracy of the inference results, the end device can perform subsequent inference based on the intermediate results fed back from the cloud server, thus improving the quality of the end device's inference results. Moreover, in scenarios with a large number of inference output characters, a large amount of incremental inference can be deployed on the end device, greatly reducing the pressure on the cloud side and significantly improving the cloud side's throughput. This effectively achieves both improved inference system throughput and high-quality inference results while shortening the initial inference latency.

[0101] The inference method provided in this application can be applied to terminal devices that work in conjunction with cloud servers. For example, the terminal device can be a mobile phone or an in-vehicle tablet, etc. The embodiments of this application do not limit the product form of the terminal device.

[0102] Figure 1 illustrates the application scenario of this application, including hardware, software, and usage scenarios. The hardware terminal device can be a mobile phone, tablet (tablet computer, in-vehicle tablet, or others), augmented reality (AR) / virtual reality (VR) device, etc. A neural processing unit (NPU) / graphics processing unit (GPU) is deployed in the terminal device to execute the solution of this application. The software can include question-and-answer interactive applications, system assistant applications, recommendation applications, or others. Usage scenarios can include question-and-answer scenarios, device control scenarios, recommendation scenarios, or others.

[0103] The application scenarios of the solutions described in the embodiments of this application are not limited to those shown in Figure 1.

[0104] The solution provided in this application can be applied to the inference system illustrated in Figure 2a. As shown in Figure 2a, the inference system includes a cloud server 201 and a terminal device 202.

[0105] The cloud server 201 is communicatively connected to the terminal device 202. The cloud server 201 deploys a large-scale LLM (Limited Language Management) to support intelligent inference services and provides services to the terminal device through a network request interface. The terminal device 202 deploys a smaller-scale LLM and serves as the human-computer interaction interface for users to access intelligent question answering, allowing interaction through clicks, touch, buttons, voice, gestures, and other methods.

[0106] Terminal device 202 provides inference software or a platform to the user. After the user inputs a question, document, or other content into the inference software or platform, terminal device 202 converts the input into prompts and sends them to cloud server 201. Cloud server 201 then performs the first X inferences (including one full inference and X-1 incremental inferences).

[0107] After performing the first X inferences, cloud server 201 sends the obtained first X lexical units and intermediate results to terminal device 202. These intermediate results include one or more of the following: the first ID sequence of the lexical units obtained by the cloud server from the full inference input of the first character; and the KV cache of the first Y lexical units among the first X lexical units obtained by the cloud server inference. Then, terminal device 202 can output the first X lexical units to the user and perform inference based on the intermediate results.

[0108] As shown in Figure 2a, the cloud server 201 may include a cloud-side processing module 2011 and a cloud-side LLM inference service 2012. The cloud-side processing module 2011, upon receiving a request from the client, controls the cloud-side LLM inference service module 2012 to perform inference only on the first X characters; and obtains the intermediate results of the cloud LLM inference on the first X characters, and distributes them as content to the cloud client. The cloud-side LLM inference service 2012 is used to perform inference after inputting a prompt word.

[0109] Furthermore, as shown in Figure 2a, the terminal device 202 includes an end-side control module 2021, an end-side inference module 2022, and a communication module 2023. Wherein:

[0110] The edge control module 2021 is used to interact with the user, initiate requests as needed and receive results from the cloud-assisted terminal. After receiving the content from the cloud-assisted terminal, it transfers it to the edge inference module 2022 for edge LLM inference.

[0111] The edge-side inference module 2022 is used to perform inference based on input prompts and intermediate results from the cloud-assisted terminal.

[0112] The communication module 2023 is used to connect to the Internet via wired or wireless communication to interact with the cloud server 201.

[0113] Of course, the internal architecture of the terminal device 202 can be configured according to actual needs. Figure 2a is only an example and does not constitute a specific limitation.

[0114] The solution provided in this application can be applied to the inference system illustrated in Figure 2b. As shown in Figure 2b, the inference system includes a server 203 and a client 204.

[0115] In one possible implementation, the server 203 and client 204 are implemented in software. For example, the server 203 could be an application, cloud service, or other form used to provide server functionality. The client 204 could be an application, platform, or other form used to provide inference services.

[0116] As shown in Figure 2b, the server 203 includes a cloud processing module 2031, a cloud inference service 2032, and a communication module 2033. The client 204 includes a user interface 2041, a control module 2042, an inference module 2043, and a communication module 2044.

[0117] Specifically, as shown in Figure 2b, the client 204 interacts with the user through the user interface 2041 to obtain the user's input. The control module 2042 sends the prompt words corresponding to the user's input to the communication module 2033 of the server 203 via the communication module 2044. The communication module 2033 transmits the received prompt words to the cloud processing module 2031, which controls the cloud inference service 2032 to perform the first X characters inference. Finally, the cloud processing module 2031 sends the first X characters and intermediate results to the communication module 2044 via the communication module 2033. The communication module 2044 transmits the first X characters and intermediate results to the control module 2042. The control module 2042 displays the first X characters through the user interface 2041 and sends the intermediate results to the inference module 2043 to perform subsequent inference to obtain the remaining characters.

[0118] Please refer to Figure 3, which is a structural schematic diagram of a terminal device provided in this application. As shown in Figure 3, the terminal device includes: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a power management module 140, an antenna, a wireless communication module 160, an audio module 170, a speaker 170A, a speaker interface 170B, a microphone 170C, a sensor module 180, buttons 190, an indicator 191, a display screen 192, a camera 193, etc. The sensor module 180 may include sensors such as a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and an ambient light sensor.

[0119] The structure illustrated in this embodiment does not constitute a specific limitation on the terminal device. In other embodiments, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0120] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0121] In this embodiment, the processor 110 is used to execute the method provided in this application to adjust the loudness of the acquired audio resources.

[0122] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0123] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a USB interface, etc.

[0124] The power management module 140 is used to connect to a power source. The power management module 140 can also be connected to the processor 110, internal memory 121, display screen 192, camera 193, and wireless communication module 160, etc. The power management module 140 receives power input and supplies power to the processor 110, internal memory 121, display screen 192, camera 193, and wireless communication module 160, etc. In some embodiments, the power management module 140 can also be located within the processor 110.

[0125] The wireless communication function of the terminal device can be implemented through an antenna and a wireless communication module 160. The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0126] The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via an antenna, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, perform frequency modulation and amplification on them, and then convert them into electromagnetic waves for radiation via the antenna. In some embodiments, the antenna of the terminal device and the wireless communication module 160 are coupled, enabling the terminal device to communicate with networks and other devices via wireless communication technology.

[0127] In this embodiment, the terminal device can communicate with the cloud server via the wireless communication module 160 and the antenna.

[0128] The terminal device implements display functions through a GPU, a display screen 192, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 192 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0129] The display screen 192 is used to display text, images, and videos, etc. The display screen 192 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0130] In this embodiment, the display screen 192 is used to display the interface provided by the terminal device, which can be used for user interaction or other functions.

[0131] The terminal device can implement shooting functions through an ISP, camera 193, video codec, GPU, display 192, and application processor. The ISP is used to process the data fed back by the camera 193. In some embodiments, the ISP can be located in the camera 193.

[0132] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device may include one or N cameras 193, where N is a positive integer greater than 1. For example, this embodiment does not limit the position of the cameras 193 on the terminal device.

[0133] Alternatively, the terminal device may not include a camera, meaning the aforementioned camera 193 is not integrated into the terminal device (e.g., a television). The terminal device can connect an external camera 193 via an interface (e.g., USB interface 130). This external camera 193 can be secured to the terminal device using an external fastener (e.g., a camera holder with a clip). For example, the external camera 193 can be fixed to the edge of the terminal device's display screen 192, such as the upper edge, using an external fastener.

[0134] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, speech recognition, and text understanding.

[0135] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, text, images, and video files can be saved on the external storage card.

[0136] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of the terminal device by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the terminal device (such as audio data, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0137] The terminal device can implement audio functions through an audio module 170, a speaker 170A, a microphone 170C, a speaker interface 170B, and an application processor. Examples include music playback and recording. In this application, the microphone 170C can be used to receive voice commands issued by the user to the terminal device. The speaker 170A can be used to provide feedback to the user on the terminal device's decision commands.

[0138] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, audio module 170 may be located in processor 110, or some functional modules of audio module 170 may be located in processor 110. Speaker 170A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Microphone 170C, also called a "microphone" or "microphone," is used to convert sound signals into electrical signals.

[0139] The speaker jack 170B is used to connect wired speakers. The speaker jack 170B can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0140] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. The terminal device can receive button input and generate key signal inputs related to user settings and function control of the terminal device.

[0141] Indicator 191 can be an indicator light, which can be used to indicate whether the terminal device is in a powered-on, standby, or powered-off state. For example, an indicator light that is off indicates that the terminal device is powered off; an indicator light that is green or blue indicates that the terminal device is powered on; and an indicator light that is red indicates that the terminal device is in a standby state.

[0142] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. It may have more or fewer components than those shown in Figure 3, may combine two or more components, or may have different component configurations. For example, the terminal device may also include components such as speakers. The various components shown in Figure 3 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing or application-specific integrated circuits.

[0143] Please refer to Figure 4, which is a schematic diagram of the structure of a server provided in this application. The server 40 can be the cloud server 201 shown in Figure 2a.

[0144] As shown in Figure 4, server 40 may include processor 4010, bus 4020, memory 4030, and communication interface 4040. Processor 4010, memory 4030, and communication interface 4040 are connected via bus 4020.

[0145] It should be understood that in this embodiment, the processor 4010 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0146] The processor 4010 may also be a GPU, NPU, microprocessor, ASIC, or one or more integrated circuits used to control the execution of the program in this application.

[0147] The communication interface 4040 is used to enable communication between the server 40 and external devices or components.

[0148] Bus 4020 may include a pathway for transferring information between the aforementioned components (such as processor 4010 and memory 4030). In addition to a data bus, bus 4020 may also include a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as bus 4020 in the diagram. Bus 4020 may be a peripheral component interconnect express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. Bus 4020 can be divided into address bus, data bus, control bus, etc.

[0149] As an example, server 40 may include multiple processors. A processor may be a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or computing units used to process data (e.g., computer program instructions).

[0150] It is worth noting that Figure 4 only shows an example of server 40 including one processor 4010 and one memory 4030. Here, processor 4010 and memory 4030 are used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business needs.

[0151] The memory 4030 can be a pool of volatile memory or a pool of non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0152] For example, the processor 4010 can perform the following functions by running or executing software programs and / or modules stored in the memory 4030:

[0153] Receive a prompt word from the end device; perform full inference and the first X-1 incremental inferences on the prompt word to obtain the first X lexical units; send the first X lexical units and intermediate results to the end device so that the end device can perform inference based on the intermediate results. The intermediate results include one or more of the following: the first ID sequence of the lexical units input to the full inference on the cloud server; and the KV cache of the first Y lexical units among the first X lexical units obtained by the cloud server inference. X is greater than or equal to 1. Y is greater than or equal to 1 and less than or equal to X. Full inference is used to obtain the first character.

[0154] The reasoning method provided in this application will now be described in detail with reference to the accompanying drawings.

[0155] Figure 5 is a flowchart illustrating the inference method provided in this application. This method can be applied to the communication process between a terminal device with inference capabilities and a cloud server. The terminal device can be the terminal device 202 in Figure 2a, and the cloud server can be the cloud server 201 in Figure 2a.

[0156] It should be noted that the solution provided in this application can be executed by an end device, or by an inference application or inference platform running on the end device. The following embodiments are described using the execution of the end device as an example, and do not constitute a specific limitation.

[0157] As shown in Figure 5, the reasoning method provided in this application embodiment may include the following steps:

[0158] S501: The terminal device sends the prompt word corresponding to the input content to the cloud server.

[0159] The input content refers to all content entered by the user and used for reasoning, which is obtained by the terminal device.

[0160] For example, the terminal device can obtain the document 1 uploaded and dragged by the user from the human-computer interaction interface, as well as the question "Summarize this document" for that document. Then, the corresponding input content would be <document, "Summarize this document">.

[0161] Specifically, the terminal device obtains the user's input through the provided human-computer interaction interface. The text of the input is then used as candidate suggestions.

[0162] In one possible implementation, the terminal device can directly use the candidate prompt words as the prompt words corresponding to the input content.

[0163] Another possible implementation is that the terminal device can preprocess the candidate prompt words to obtain the prompt words corresponding to the input content. Preprocessing can include one or more of the following operations: de-identification, compression, or encryption.

[0164] Among them, the desensitization operation is used to replace sensitive words with corresponding characters in the desensitization rules. The desensitization rules include one or more sensitive words and corresponding characters. The content of the desensitization specifications can be configured according to actual needs, and the present application does not limit the content of the desensitization rules.

[0165] Exemplarily, the input content is <document, "Summarize the document">, and the original prompt word corresponding to this input content is: all the text content in the document + "Summarize the document". The desensitization operation can be: replace sensitive information such as mobile phone numbers and email addresses in the prompt word with specific placeholders.

[0166] Further optionally, the terminal device can also send a transmission feature to the cloud server, and the transmission feature is used to indicate the amount of data that can be transmitted between the terminal device and the cloud server. So that the cloud server can determine the amount of data of the interaction content according to the transmission feature between the two.

[0167] In a possible implementation manner, the transmission feature includes the network state between the terminal device and the cloud server, and / or the user level corresponding to the terminal device. Among them, the network state can be the connection method, such as WIFI connection. The user level can be the level in the level contract signed by the user and the service provider.

[0168] Exemplarily, assume that the terminal device is currently connected to the cloud server via the WIFI method, and the user level of the logged-in terminal device is a VIP user. The transmission feature can be <WIFI, VIP>.

[0169] Further optionally, the terminal device can also send specification information to the cloud server, and the specification information is used to indicate the cloud-side model specification corresponding to the feature of the input content. Among them, the feature of the input content can be the data volume of the input content, the number of characters of the input content, or other features, and the embodiments of the present application do not specifically limit the features of the input content.

[0170] Exemplarily, relevant rules can be configured, and the rules include the corresponding relationship between different features and model specifications. The terminal device can obtain the corresponding model specification in the rules according to the feature of the input content and send it to the cloud server. So that the cloud server can perform reasoning according to the requested model specification and assist the terminal. For example, for the feature of the input content <document, "Summarize the document">, the corresponding model specification is: tttB z series. Among them, the z series is a series of cloud models, and this series includes multiple specifications: tB, ttB, tttB, etc. Then, the terminal device can send the specification information to the cloud server as: <tttB, z series>.

[0171] In a possible implementation manner, the content sent by the terminal device to the cloud server described in S501 can be sent through one or more request bodies, which is not limited.

[0172] Further optionally, after the terminal device executes S501, it waits for feedback from the cloud server. The terminal device can make full use of this time period to segment the original prompt word (convert the prompt word into an input recognizable by the local LLM, that is, an ID sequence; for example, "abstract" corresponds to 10002), and perform necessary initializations for local LLM inference, such as loading model weights and other operations, to prepare for inference.

[0173] S502. The cloud server receives the prompt word from the terminal device.

[0174] Specifically, the prompt word received by the cloud server in S502 is the prompt word sent by the terminal device in S501, which will not be elaborated here.

[0175] In a possible implementation, if the prompt word obtained by the terminal device in S501 through preprocessing, then in S502, the cloud server receives the preprocessed prompt word from the terminal device. The preprocessing may include compression operations or encryption operations.

[0176] S503. The cloud server performs full-scale inference and the first X - 1 incremental inferences on the prompt word to obtain the first X word tokens.

[0177] Among them, full-scale inference is used to obtain the first character of the answer corresponding to the prompt word for the above input content. One incremental inference is used to obtain the character after the first character in the answer.

[0178] It should be understood that if the prompt word received in S502 is a preprocessed prompt word, S503 is specifically implemented as: performing the inverse operation of the preprocessing on the prompt word, and then performing full-scale inference and the first X - 1 incremental inferences. Among them, the inverse operation of the compression operation is the decompression operation, and the inverse operation of the encryption operation is the decryption operation, and the specific process will not be elaborated here.

[0179] In a possible implementation, the cloud server also receives the specification information sent by the terminal device. S503 is specifically implemented as: performing full-scale inference and the first X - 1 incremental inferences on the prompt word according to the model specification indicated by the specification information. For example, the cloud server receives the prompt word corresponding to the input content <document, "Summarize this document">, and also receives the specification information: <tttB, z series>. The cloud server then runs the model of the z series tttB specification to perform inference on the prompt word corresponding to the input content <document, "Summarize this document">.

[0180] Further, after obtaining the prompt word, first segment it to obtain word tokens, and input the ID sequence of the word tokens into the LLM for inference. For the process of the LLM model performing inference, reference can be made to the foregoing content, which will not be elaborated here.

[0181] S504. The cloud server sends the first X tokens and intermediate results to the end device. The intermediate results include one or more of the following: the first ID sequence of the tokens input by the cloud server in full inference, and the KV cache of the first Y tokens among the first X tokens obtained by the cloud server inference.

[0182] Where X is greater than or equal to 1; Y is greater than or equal to 1 and less than or equal to X. The values ​​of X and Y can be configured according to actual needs.

[0183] Specifically, cloud LLMs and edge LLMs differ in specifications (generally manifested in the number of parameters), but more specifically in the number of layers and the size of the key-value cache within each layer. For example, a cloud xxB might have dozens of layers, with each layer's key-value cache containing S*D positions. Here, S is the prompt word length of the cloud LLM (i.e., the total number of characters in the document and user query), and D is a hyperparameter for a specific layer, indicating how many positions are used to represent a single character in the input. For instance, if S = prompt word length 100 and D = 200, each layer's key-value cache has 20,000 positions (note that the size of the key-value cache may vary between layers). Similarly, edge LLMs, being smaller, have only 10-20 layers, with each layer's key-value cache only having a few thousand positions. In S504, the cloud server needs to first determine the intermediate results, which are adapted to the edge device and can be directly used by it. The intermediate results adapted to the edge device include the number of key-value cache layers included in the intermediate results for one inference operation, up to a maximum of Q layers in the edge LLM. The intermediate results include a KV cache for one layer, which is at most the number of KV cache locations for that layer in the endpoint LLM.

[0184] For example, disregarding the amount of data transmitted, the number of KV cache layers included in the intermediate result determined by the cloud server for one inference operation is equal to the number of layers Q of the endpoint LLM. The KV cache of one layer included in the intermediate result is equal to the number of KV cache locations of that layer in the endpoint LLM. In this way, the endpoint can directly use the intermediate result to perform the Yth incremental inference (i.e., the Y+1th inference of the overall inference).

[0185] For example, the process by which a cloud server determines intermediate results may include the following steps:

[0186] Step 1: In the LLM of the cloud server, determine the first i consecutive layers as sparseable layers.

[0187] The sparseable layer includes the ID sequence corresponding to the prompt words. The number of parameters in the j-th layer of the first i layers of the LLM in the cloud server is greater than or equal to the number of parameters in the j-th layer of the first i layers of the LLM in the end device, where j iterates from 1 to i. The number of parameters in the (i+1)-th layer of the LLM in the cloud server is less than the number of parameters in the i-th layer of the LLM in the end device, or i equals Q. In other words, the determined sparseable layer can be a layer with a size greater than or equal to the end-side LLM, thus effectively assisting the end-side and supporting sparsity. The determined sparseable layer is continuous and starts from the ID sequence.

[0188] Figure 6 illustrates the process of determining sparseable layers in a cloud server. As shown in Figure 6, the cloud server compares its local LLM with the client-side LLM, layer by layer (including the ID sequence input before layer 1). A layer is considered sparseable if the KV cache size (which can be the number of parameters) of the cloud LLM's layer j is greater than or equal to the KV cache size of the client LLM's layer j. During the comparison, if the KV cache size of the cloud-side layer k is smaller than the KV cache size of the client LLM's layer k, then layers 1 through k-1 are considered sparseable layers. Alternatively, if the number of determined sparseable layers equals the number of layers in the client LLM, then the result determines the sparseness.

[0189] Step 2: Reduce the number of parameters in the KV cache of each sparseable layer performing the target inference to obtain the target KV cache.

[0190] The purpose of step 2 is to reduce the number of key-value caches in one layer to the same number required in the endpoint LLM. The objects to be reduced in step 2 vary depending on the value of Y.

[0191] In one possible implementation, Y is 1, meaning that the intermediate results of the cloud server only provide KV cache for full inference. In step 2, the number of parameters in the KV cache of each sparse layer that performs full inference can be reduced to obtain the target KV cache of each sparse layer that performs full inference. The number of parameters in the target KV cache of a sparse layer that performs full inference is equal to the number of parameters in the same position layer of the LLM in the end device.

[0192] In another possible implementation, Y is greater than 1, meaning that the intermediate results of the cloud server assistant not only provide KV cache for full inference but also KV cache for partial incremental inference. In step 2, the number of parameters in the KV cache of each sparse layer that performs full inference and the first Y-1 incremental inferences can be reduced to obtain the target KV cache of each sparse layer that performs full inference and the first Y-1 incremental inferences. The number of parameters in the target KV cache of a sparse layer that performs full inference and the first Y-1 incremental inferences is equal to the number of parameters in the same position layer of the LLM in the end device.

[0193] After step 2, if the amount of data transmitted is not considered, or if the network at the end is good (such as a WIFI network), the target KV cache and the input ID sequence can be directly determined as intermediate results.

[0194] However, the target KV cache and input ID sequence data volume are very large, and the transmission characteristics between the end and the cloud may not support the transmission (e.g., when the end network conditions are poor, such as 3G). In this case, the target KV cache and input ID sequence can be sparsified to reduce the transmission volume. However, correspondingly, the latency of the inference process on the end also increases. Accordingly, step 3 below needs to be performed after step 2.

[0195] Step 3: Determine intermediate results from the target KV cache whose data volume is less than or equal to the target data volume.

[0196] In step 3, an intermediate result is determined by using a sparse approach, where the data volume is less than or equal to the target data volume. The purpose of sparsity is to reduce the data volume of the cloud-assisted content. The cloud server can determine the target data volume corresponding to the transmission characteristics in the first rule based on the transmission characteristics between the cloud server and the terminal device; the intermediate result where the data volume is less than or equal to the target data volume is determined by sparsity.

[0197] The first rule refers to the pre-configured data volume corresponding to different transmission characteristics. The content of the first rule can be configured according to actual needs, and this embodiment of the application does not limit it.

[0198] For example, transport characteristics are used to indicate the amount of data that can be transmitted between the endpoint and the cloud. For instance, transport characteristics may be used to indicate the network status between the endpoint device and the cloud server, and / or the user level corresponding to the endpoint device.

[0199] In one possible implementation, the cloud server can determine the network status between the two devices by receiving content sent by the receiving device, and then receive the user level from the receiving device.

[0200] In another possible implementation, the cloud server can receive transmission characteristics sent by the end device.

[0201] After determining the target data volume, the cloud server can determine the final intermediate result using a sparse approach. In practical applications, the specific sparsity method can be configured according to actual needs, and this application embodiment does not limit this. The following describes two sparsity methods provided in this application.

[0202] Sparseness technique 1: Layer reduction. Reduce the number of sparseable layers (1 to i) to a total of sparseable layers (1 to i-1). This means reducing the transmission of one or more key-value buffers, with the reduced transmission layers handled by the endpoint for inference.

[0203] Sparsity Method 2: Sparse Content. Group Query Attention (GQA) is used for compression and filtering. That is, only a portion of the original key-value cache / ID sequence is retained as an intermediate auxiliary result.

[0204] In one possible implementation, the compression filtering method in sparse method 2 can be: removing one every M, removing one every M, and so on. M can be configured according to actual needs, and this embodiment of the application does not limit it.

[0205] In another possible implementation, the compression filtering method in sparsity approach 2 could be as follows: for the ID sequence corresponding to the prompt words, identify and retain the most critical IDs. For example, the KV cache at each layer can be averaged, and then the ID sequences can be sorted by weight to filter them, with a threshold configured to retain important key IDs.

[0206] In one possible implementation, after sparsifying the ID sequence corresponding to the prompt word, the key-value cache corresponding to the pruned ID also needs to be pruned to ensure the accuracy and feasibility of the inference process.

[0207] In practical applications, multiple sparsity techniques can be used in combination to make the intermediate result less than or equal to the target data size.

[0208] In one possible implementation, in the scenario where Y equals 1, after obtaining the target KV cache of the sparse layer for full inference, the intermediate result whose data volume is less than the target data volume is determined from the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparse layer for full inference.

[0209] For example, if the word ID sequence corresponding to the prompt word and the data volume of the target KV cache of the sparse layer for performing full inference are less than or equal to the target data volume, the word ID sequence corresponding to the prompt word and the target KV cache of the sparse layer for performing full inference are determined as intermediate results. The first ID sequence is the word ID sequence corresponding to the prompt word.

[0210] For example, if the lexical ID sequence corresponding to the prompt word and the data volume of the target KV cache of the sparseable layer performing full inference are greater than the target data volume, a sparsification operation is performed on the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparseable layer performing full inference to obtain an intermediate result with a data volume less than or equal to the target data volume. The sparsification operation includes one or more of the following operations: deleting one or more layers of KV cache, or sparsifying the KV cache in one or more layers. It should be understood that the sparsification operation can be one or more of the above-mentioned sparsification method 1 and sparsification method 2. The first ID sequence is the sparsified content of the lexical ID sequence corresponding to the prompt word.

[0211] In one possible implementation, in scenarios where Y is greater than 1, after obtaining the target KV cache of each sparse layer that performs full inference and the first Y-1 incremental inferences, the intermediate result whose data volume is less than or equal to the target data volume is determined from the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparse layer that performs full inference and the first Y-1 incremental inferences.

[0212] For example, if the token ID sequence corresponding to the prompt word and the data volume of the target KV cache of the sparse layer during full inference and the first Y-1 incremental inferences are less than or equal to the target data volume, the token ID sequence corresponding to the prompt word and the target KV cache of the sparse layer during full inference and the first Y-1 incremental inferences are determined as intermediate results. The first ID sequence is the token ID sequence corresponding to the prompt word.

[0213] For example, if the token ID sequence corresponding to the prompt word, and the data volume of the target KV cache of the sparseable layer after performing full inference and the first Y-1 incremental inferences, are greater than the target data volume, content sparsification is performed on the token ID sequence corresponding to the prompt word, and the target KV cache of the sparseable layer after the first Y-2 incremental inferences (i.e., the above-mentioned sparsification method 2), and / or, sparsification operation is performed on the target KV cache of the sparseable layer of the Y-1th incremental inference, to obtain an intermediate result in which the data volume is less than or equal to the target data volume. The sparsification operation includes one or more of the following operations: deleting one or more layers of KV cache, or sparsifying the KV cache in one or more layers. It should be understood that the sparsification operation can be one or more of the above-mentioned sparsification method 1 and sparsification method 2. The first ID sequence is the content after sparsification of the token ID sequence corresponding to the prompt word.

[0214] It should be understood that in scenarios where Y is greater than 1, only the target KV cache of the sparseable layer in the (Y-1)th incremental inference is subjected to the layer trimming operation (sparse means 1) to ensure the integrity of the target KV cache layer in other inference processes. This makes it easy for the end side to only continue to execute the (Y-1)th incremental inference without having to execute the full inference and the first (Y-2)th incremental inference.

[0215] In one possible implementation, the sparse process can remove the KV cache of all sparseable layers, and the intermediate result is only the sparsed ID sequence.

[0216] Figure 7 illustrates the sparse intermediate result scheme. As shown in Figure 7, there are i sparse layers. Using sparsity method 1, layers 2 to i are removed, retaining only the ID sequence and the KV cache of layer 1. Then, sparsity method 2 is applied to layer 1 and the ID sequence respectively. For the KV cache of layer 1, one KV is retained for every two KVs. The ID sequence is filtered based on the KV cache values ​​of each layer (average of the KV cache values ​​of each layer, then sorted by weight to filter the ID sequences, retaining important key IDs). Subsequently, the intermediate result from the cloud assistant is the sparse ID sequence and the sparse KV cache of layer 1. After receiving the intermediate result from the cloud assistant, the terminal performs the transformation and recovery of layer 1, and uses the ID sequence to continue the inference from layer 2, completing the entire full inference process.

[0217] S505: The terminal device receives the first X lexical units and intermediate results obtained by the cloud server through reasoning on the prompt words.

[0218] It should be understood that the content received by the S505 mid-end device is the same as the content sent by the cloud server in the S504.

[0219] S506, the terminal device outputs the first X words.

[0220] For example, the terminal device can use speed control to output the first X words one by one, avoiding excessively long word gaps and achieving a smooth effect. This application embodiment does not specifically limit the method by which the terminal device outputs the first X words. The speed of the speed control can be set according to actual needs, and this application embodiment does not limit it in this regard.

[0221] Furthermore, the terminal device can execute the S507 process while executing S506.

[0222] S507: The terminal device performs reasoning based on the intermediate results to obtain and output the remaining word units in the answer corresponding to the prompt word.

[0223] Since the intermediate results include the first ID sequence and / or KV cache, the content of the cloud assistant can be one or more of the following two methods.

[0224] Figure 8 is a schematic diagram of a cloud-based assistance scenario. As shown in Figure 8, the cloud-based assistance includes two assistance methods:

[0225] Method 1: Input content for each inference by the assistant side through the first ID sequence.

[0226] Method 2: Through the KV cache of the i-th layer, the terminal side executes the KV cache content of the i-th layer inference.

[0227] After the terminal device obtains the intermediate result of the cloud assistant terminal, it performs corresponding processing according to the content of the intermediate result.

[0228] In a possible implementation, if the number of layers of the KV cache in the intermediate result is the same as the number of layers of the terminal-side LLM, and the number of KV caches in each layer is the same as the number of KV cache positions in the same layer of the terminal-side LLM, the terminal device can directly use the intermediate result and then perform the next incremental inference not included in the intermediate result.

[0229] In another possible implementation, if the number of layers in the intermediate result is less than the number of layers of the terminal-side LLM, but the number of KV caches in each layer is the same as the KV cache positions in the same layer of the terminal-side LLM, the terminal device can continue to perform the inference of the layers not included. Then perform the next incremental inference.

[0230] In yet another possible implementation, if the number of KV caches in each layer in the intermediate result is less than the number of KV cache positions in the same layer of the terminal-side LLM, the terminal device needs to first perform filling to serve as the KV cache for the terminal-side LLM to perform the inference of the corresponding layer. Then continue to perform the inference of the layers not included. Then perform the next incremental inference.

[0231] For example, if there are a total of A positions to be filled in a certain layer of the terminal LLM, and there are only B (B < A) KV caches in this layer of the intermediate result of the cloud assistant terminal after sparsification, fill the invalid positions (for example, fill with 0, or others, not limited), to form the values of the A positions required for this layer of the terminal LLM.

[0232] In another possible implementation, for the ID sequence in the intermediate result, the terminal device directly uses it as the input of the LLM.

[0233] Combined with the filling scheme, the above two assisting methods are exemplarily described as follows:

[0234] Assisting method 1: The ID sequence after tokenization of the original prompt is A, such as […, 10000, 15000, 10001, 20001, 10002, 30000], corresponding to […, right, this, document, perform, abstract, summarize]), and the sparsified ID sequence of the cloud assistant terminal is B, such as […, 10002, 30000], corresponding to […, abstract, summarize]. Then the input of the terminal LLM can be the filled B, such as […, 0, 0, 0, 0, 10002, 30000]. Or, the input of the terminal LLM can be [10002, 30000] to reduce the scale of data processing.

[0235] Method 2: The KV cache at the e-th layer of the cloud-assisted LLM has a total of R positions (R is the sum of the number of characters in the document and the number of characters in the question "summarize the document"). The intermediate results include R / 2 positions of the KV cache at the e-th layer. These R / 2 positions are used to restore the R positions of the e-th layer executed by the cloud-assisted LLM. One restoration and filling scheme is GQA, where one position on the cloud-assisted end corresponds to multiple positions in the e-th layer of the cloud-assisted LLM. For example, if the R / 2 positions on the cloud-assisted end are [1,2,...], the restored R positions are [1,1,2,2,...].

[0236] Specifically, the process by which the terminal device performs reasoning based on the intermediate results is related to the content of the intermediate results, and will be described in different cases below.

[0237] Case 1: The intermediate result contains the first ID sequence.

[0238] In Case 1, due to sparse operations on the cloud side or transmission losses, the intermediate result does not include the KV cache, but only the first ID sequence. The terminal device executes S507 specifically as follows: based on the first ID sequence, it performs full inference and subsequent incremental inference until the stopping condition is met. The inference results of the Xth and subsequent incremental inferences are used as the remaining tokens in the answer and output. Each incremental inference is used to obtain one token after the first character of the answer. In Case 1, the cloud only assists the terminal with input content.

[0239] That is, if the intermediate result from the cloud-assisted terminal is an ID sequence, the terminal starts reasoning from level 1 of the full inference. If the cloud-assisted terminal's result is an intermediate result from levels 1 to i of the full inference, the terminal starts reasoning from level i+1 of the full inference to continue reasoning for the first character. After the first character reasoning is completed, the terminal LLM then performs subsequent incremental reasoning. Since the cloud only assists with the first character, each subsequent incremental reasoning still needs to calculate all levels one by one. After each incremental reasoning completes and outputs a character, the terminal returns it to the user.

[0240] Case 2: Y is 1, and the intermediate results include the KV cache from the first layer to the a-th layer during the full inference process performed by the cloud server. a is less than or equal to Q.

[0241] In scenario 2, the cloud-assisted terminal includes a KV cache for full inference but not for incremental inference. The terminal device executes S507 as follows:

[0242] The endpoint device first performs full inference on the cloud server using the key-value caches from layer 1 to layer a. This determines the key-value caches used by the endpoint device's LLM for performing full inference on layers 1 to a (the endpoint device does not actually perform full inference on layers 1 to a). During this determination process, if the number of key-value caches for a certain layer is less than the corresponding key-value cache position in the endpoint device's LLM, it is padded; if the number of key-value caches for a certain layer is equal to the corresponding key-value cache position in the endpoint device's LLM, it is used directly. A specific example is shown below:

[0243] For example, the endpoint performs a first inference on the cloud server. This first inference can be either a full inference or an incremental inference. Based on the key-value (KV) cache of the i-th layer of the cloud server's first inference, the endpoint device determines the KV cache of the i-th layer of the endpoint's LLM inference. This includes: if, in the intermediate results, the total number of parameters in the KV cache of the i-th layer of the cloud server's first inference is the same as the number of parameter positions in the KV cache of the i-th layer of the LLM's inference, then the KV cache of the i-th layer of the cloud server's first inference is used as the KV cache of the i-th layer of the endpoint's LLM inference. Here, i is any value from 1 to Q. If, in the intermediate results, the total number of parameters in the KV cache of the i-th layer of the cloud server's first inference is less than the number of parameter positions in the KV cache of the i-th layer of the LLM's inference, then the KV cache of the i-th layer of the cloud server's first inference is padded and used as the KV cache of the i-th layer of the LLM's inference.

[0244] The above filling includes: filling a value in the KV cache of the i-th layer of the cloud server performing the first inference, and filling multiple parameter positions in the KV cache of the i-th layer of the LLM performing inference.

[0245] Then, the terminal device, based on the reference ID sequence and the KV cache of layers 1 to 'a' used by the LLM for performing full inference, performs the Y-th incremental inference and subsequent incremental inferences until the stopping condition is met. The inference results of the X-th and subsequent incremental inferences are used as the remaining tokens in the answer and output. Wherein, if the intermediate result includes the first ID sequence, the reference ID sequence is the first ID sequence. If the intermediate result does not include the first ID sequence, the reference ID sequence is the second ID sequence of the tokens corresponding to the prompt words. The second ID sequence is obtained through conversion, which is not limited in this embodiment.

[0246] Specifically, the end device performs the Yth incremental inference and subsequent incremental inference processes based on the reference ID sequence and the KV cache of the first to the ath layers used by the LLM to perform full inference. The end device performs different operations according to the relationship between a and the layer number Q of the end-side LLM.

[0247] When a is Q, the terminal device will input the reference ID sequence into the terminal-side LLM, and based on the determined terminal-side KV cache for full inference of the LLM from the first layer to the a-th layer, perform the Y-th incremental inference on the terminal side.

[0248] When a is less than Q, the reference ID sequence is input into the edge LLM. Based on the KV cache of layers 1 to 'a' used by the edge LLM for full inference, the inference of layers a+1 to 'Q' is performed, and the KV cache of layers a+1 to 'Q' used by the edge LLM for full inference is obtained. Then, the edge device performs the Y-th incremental inference based on the reference ID sequence and the KV cache of layers 1 to 'Q' used by the edge LLM for full inference.

[0249] In LLM inference, the ID sequence only needs to be input once, and the model caches the ID sequence for subsequent inference. However, if the ID sequence is modified during inference (e.g., through correction or other means), the ID sequence needs to be input into the model again.

[0250] Case 3: Y is greater than 1. The intermediate results include the KV cache of the first layer to the Q layer during the full inference and the first Y-2 incremental inference processes performed by the cloud server, as well as the KV cache of the first layer to the b layer during the Y-1 incremental inference performed by the cloud server, where b is less than or equal to Q.

[0251] In scenario 3, the key-value cache included in the cloud-assisted terminal not only includes the key-value cache for full inference but also the key-value cache for incremental inference. The execution process of S507 on the terminal device varies depending on the relationship between b and the number of layers Q in the terminal-side LLM, as detailed below:

[0252] When b is Q, the endpoint device performs full inference and the first Y-1 incremental inference processes using the KV cache of layers 1 to Q on the cloud server. It determines the endpoint LLM used for performing full inference and the first Y-1 incremental inference processes using the KV cache of layers 1 to Q (in reality, full inference and the first Y-1 incremental inference processes using the first Y-Q layers are not performed). Then, the reference ID sequence is input into the endpoint LLM. Based on the determined endpoint LLM's KV cache used for performing full inference and the first Y-1 incremental inference processes using the first Y-Q layers, the endpoint performs the Y-th incremental inference and subsequent incremental inference until the stopping condition is met. The inference results of the X-th and subsequent incremental inferences are used as the remaining tokens in the above answer and output.

[0253] When b is less than Q, the end device performs full inference and the first Y-2 incremental inferences using the KV cache of layers 1 to Q based on the cloud server. It then determines that the end-side LLM is used to perform the full inference and the first Y-2 incremental inferences using the KV cache of layers 1 to Q (in reality, the full inference and the first Y-2 incremental inferences using the first Y-2 incremental inferences are not performed). Then, the end device performs the (Y-1)th incremental inference using the KV cache of layers 1 to b based on the cloud server. It then determines that the end-side LLM is used to perform the (Y-1)th incremental inference using the KV cache of layers 1 to b (in reality, the (Y-1)th incremental inference using the first Y-1 incremental inference is not performed). Next, the terminal device inputs the reference ID sequence into the terminal-side LLM. Based on the KV cache of layers 1 to Q used in the terminal-side LLM for full inference and the first Y-2 incremental inferences, and the KV cache of layers 1 to b used in the Y-1 incremental inference process, the terminal-side device performs inference from layers b+1 to Q in the Y-1 incremental inference, and obtains the KV cache of layers b+1 to Q used in the Y-1 incremental inference. Finally, based on the reference ID sequence and the KV cache of layers 1 to Q used in the terminal-side LLM for full inference and the first Y-1 incremental inferences, the terminal device performs the Y-th incremental inference and subsequent incremental inferences until the stopping condition is met. The inference results of the Xth and subsequent incremental inferences are used as the remaining tokens in the above answer and output.

[0254] Furthermore, in S507, the intermediate device takes the inference results of the Xth and subsequent incremental inferences as the remaining tokens in the above answer and outputs them, continuing the action of S506 to ensure the continuity of the inference results. Then, it continues to execute subsequent incremental inferences until the LLM stopping condition is met. The stopping condition can be that the inference result is a preset stop character, or that the number of inferences reaches a preset threshold, or something else.

[0255] The solution provided in this application employs an edge-cloud collaborative approach for inference. The cloud server performs the inference of the first X characters, including the first character, thus reducing the latency of full inference. The cloud server feeds back intermediate results during the inference process. Due to the short latency of the first character inference on the cloud server, the latency of the first character inference is significantly reduced. Furthermore, because the cloud server is large-scale and has high inference accuracy, the edge device can perform subsequent inference based on the intermediate results fed back by the cloud server, thereby improving the quality of the edge device's inference results. Moreover, in scenarios with a large number of inference output characters, a large number of incremental inferences can be deployed on the edge device, greatly reducing the pressure on the cloud device and significantly improving its throughput. This effectively achieves both reduced latency of the first character inference and improved throughput of the inference system, while ensuring high-quality edge device inference results.

[0256] The foregoing mainly describes the solution provided in this application. Accordingly, this application also provides an inference device for implementing various functions of the mid-range device or cloud server in the above method embodiments.

[0257] In some embodiments, the inference device includes hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-described functions. Those skilled in the art will readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0258] This application embodiment can divide the inference device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0259] On one hand, this application provides an inference device 90, which is used to implement the functions of the intermediate device in the above method embodiments. As shown in FIG9, the inference device 90 may include a sending unit 901, a receiving unit 902, an output unit 903, and an inference unit 904.

[0260] Specifically, the transmitting unit 901 executes operation S501 in the method illustrated in Figure 5. The receiving unit 902 executes operation S505 in the method illustrated in Figure 5. The output unit 903 executes operation S506 in the method illustrated in Figure 5. The inference unit 904 executes operation S507 in the method illustrated in Figure 5.

[0261] On the other hand, this application provides an inference device 100, which is used to implement the functions of the cloud server in the above method embodiments. As shown in FIG10, the inference device 100 may include a receiving unit 1001, an inference unit 1002, and a sending unit 1003.

[0262] The receiving unit 1001 is used to execute operation S502 in the method illustrated in FIG5. The inference unit 1002 is used to execute operation S503 in the method illustrated in FIG5. The sending unit 1003 is used to execute operation S504 in the method illustrated in FIG5.

[0263] In another aspect, embodiments of this application provide an inference system, which includes an inference device 90 and an inference device 100.

[0264] In another aspect, embodiments of this application provide an inference system, which includes a terminal device and a cloud server. An inference device 90 is deployed in the terminal device, and an inference device 100 is deployed in the cloud server.

[0265] Furthermore, embodiments of this application also provide a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the operational steps of the method in the above-described method embodiments.

[0266] Furthermore, embodiments of this application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the operational steps of the method described in the above method embodiments.

[0267] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in the computing device.

[0268] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A reasoning method, characterized in that, The method, applied to or in reasoning applications within end devices, includes: Send prompt words corresponding to the input content to the cloud server; The system receives data from the cloud server, including the first X lexical units obtained by the cloud server through reasoning on the prompt word, and intermediate results. The intermediate results include one or more of the following: the first identifier ID sequence of the lexical units input by the cloud server's full reasoning; the KV cache of the first Y lexical units among the first X lexical units obtained by the cloud server's reasoning; where X is greater than or equal to 1; Y is greater than or equal to 1 and less than or equal to X; the full reasoning is used to obtain the first character in the answer corresponding to the prompt word. Output the first X words; Based on the intermediate results, reasoning is performed to obtain the remaining words in the answer and output them.

2. The method according to claim 1, characterized in that, The intermediate result is the first ID sequence, and the reasoning based on the intermediate result includes: Based on the first ID sequence, perform full inference and subsequent incremental inference; each incremental inference is used to obtain a word after the first character.

3. The method according to claim 1, characterized in that, Y is 1, the number of layers of the large language model LLM in the terminal device is Q, and the intermediate results include the KV cache from the first layer to the ath layer during the full inference process performed by the cloud server. The reasoning based on the intermediate results to obtain and output the remaining lexical units in the answer includes: Based on the KV cache of the first to the ath layers in the full inference process performed by the cloud server, the KV cache of the first to the ath layers used by the LLM for performing the full inference is determined; Based on the reference ID sequence and the KV cache of the first to the ath layers used by the LLM to perform full inference, perform the Yth incremental inference and subsequent incremental inference until the stopping condition is met; if the intermediate result includes the first ID sequence, the reference ID sequence is the first ID sequence; if the intermediate result does not include the first ID sequence, the reference ID sequence is the second ID sequence of the word corresponding to the prompt word. The reasoning results of the incremental reasoning from the Xth time onwards are used as the remaining words in the answer and output.

4. The method according to claim 3, characterized in that, The term 'a' refers to Q, and the execution of the Y-th incremental inference, based on the reference ID sequence and the KV cache of the first to a-th layers of the LLM used for full inference, includes: The reference ID sequence is input into the LLM, and the Yth incremental inference is performed based on the KV cache of the first to the ath layers used by the LLM to perform full inference.

5. The method according to claim 3, characterized in that, The condition 'a' is less than 'Q', and the incremental inference based on the reference ID sequence and the KV cache of the first to the 'a' layers of the LLM used for full inference includes: Input the reference ID sequence into the LLM, and based on the KV cache of the LLM used to perform full inference from the first layer to the a-th layer, perform full inference from the (a+1)-th layer to the Q-th layer, and obtain the KV cache of the LLM used to perform full inference from the (a+1)-th layer to the Q-th layer; Based on the KV cache of the first to Q layers of the LLM, perform the Yth incremental inference.

6. The method according to any one of claims 1-5, characterized in that, Performing a first inference on the cloud server, where the first inference is either a full inference or an incremental inference, and determining the KV cache of the i-th layer of the LLM's inference based on the KV cache of the i-th layer of the first inference performed on the cloud server, includes: If, in the intermediate results, the total number of parameters in the KV cache of the i-th layer of the first inference performed by the cloud server is the same as the number of parameter positions in the KV cache of the i-th layer of the inference performed by the LLM, then the KV cache of the i-th layer of the first inference performed by the cloud server is used as the KV cache of the i-th layer of the inference performed by the LLM; where i is any value from 1 to Q. If, in the intermediate results, the total number of parameters in the KV cache of the i-th layer of the first inference performed by the cloud server is less than the number of parameter positions in the KV cache of the i-th layer of the LLM inference, the KV cache of the i-th layer of the first inference performed by the cloud server is filled and used as the KV cache of the i-th layer of the LLM inference.

7. The method according to claim 6, characterized in that, The filling includes: filling multiple parameter positions in the KV cache of the i-th layer of the cloud server executing the first inference with a value from the first inference layer of the LLM.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Obtain the input content; Convert the input content into candidate suggestion words; The candidate prompt words are preprocessed to obtain the prompt words; the preprocessing includes one or more of the following operations: desensitization operation, compression operation, or encryption operation; the desensitization operation is used to replace sensitive words with corresponding characters in the desensitization rules; the desensitization rules include one or more sensitive words and their corresponding characters.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Send transmission characteristics and / or specification information to the cloud server, wherein the transmission characteristics are used to indicate the amount of data that can be transmitted between the end device and the cloud server, and the specification information is used to indicate the cloud-side model specification corresponding to the characteristics of the input content.

10. A reasoning method, characterized in that, Applied to cloud servers, the method includes: Receive prompts from the terminal device; Perform full inference and the first X-1 incremental inferences on the prompt words to obtain the first X lexical units; The first X lexical units and intermediate results are sent to the terminal device; the intermediate results include one or more of the following: the first identifier ID sequence of the lexical units input by the full inference of the cloud server, the KV cache of the first Y lexical units in the first X lexical units obtained by the cloud server inference; X is greater than or equal to 1; Y is greater than or equal to 1 and less than or equal to X; the full inference is used to obtain the first character in the answer corresponding to the prompt word.

11. The method according to claim 10, characterized in that, The method further includes: Based on the transmission characteristics between the cloud server and the terminal device, the target data volume corresponding to the transmission characteristics in the first rule is determined; the transmission characteristics are used to indicate the amount of data that can be transmitted between the terminal device and the cloud server. The intermediate result is determined to have a data volume smaller than the target data volume.

12. The method according to claim 11, characterized in that, In the terminal device, the LLM is the Q layer, Y is 1, and the intermediate result of determining that the data volume is less than the target data volume includes: In the LLM of the cloud server, the first i consecutive layers are identified as sparseable layers; the sparseable layers include the ID sequence corresponding to the prompt words; the number of parameters in the j-th layer of the first i layers of the LLM in the cloud server is greater than or equal to the number of parameters in the j-th layer of the first i layers of the LLM in the terminal device, where j traverses from 1 to i; the number of parameters in the (i+1)-th layer of the LLM in the cloud server is less than the number of parameters in the i-th layer of the LLM in the terminal device, or i is equal to Q; The number of parameters in the KV cache of each of the sparseable layers performing full inference is reduced to obtain the target KV cache of each of the sparseable layers performing full inference. The number of parameters in the target KV cache of a sparseable layer performing full inference is equal to the number of parameters in the same position layer of the LLM in the end device. From the lexical ID sequence corresponding to the prompt word and the target KV cache of the sparse layer during full inference, determine the intermediate result whose data volume is less than or equal to the target data volume.

13. The method according to claim 11, characterized in that, In the terminal device, the LLM is at the Q layer, Y is greater than 1, and the intermediate result of determining that the data volume is less than the target data volume includes: In the LLM of the cloud server, the first i consecutive layers are identified as sparseable layers; the sparseable layers include the ID sequence corresponding to the prompt words; the number of parameters in the j-th layer of the first i layers of the LLM in the cloud server is greater than or equal to the number of parameters in the j-th layer of the first i layers of the LLM in the terminal device, where j traverses from 1 to i; the number of parameters in the (i+1)-th layer of the LLM in the cloud server is less than the number of parameters in the i-th layer of the LLM in the terminal device, or i is equal to Q; The number of parameters in the KV cache of each sparseable layer that performs full inference and the first Y-1 incremental inferences is reduced to obtain the target KV cache of each sparseable layer that performs full inference and the first Y-1 incremental inferences. The number of parameters in the target KV cache of a sparseable layer that performs full inference and the first Y-1 incremental inferences is equal to the number of parameters in the same position layer of the LLM in the end device. From the lexical ID sequence corresponding to the prompt word, and from the target KV cache of the sparse layer after performing full inference and the first Y-1 incremental inferences, determine the intermediate result whose data volume is less than or equal to the target data volume.

14. The method according to any one of claims 11-13, characterized in that, The method further includes: The receiving end device sends the transmission characteristics.

15. The method according to any one of claims 10-14, characterized in that, The method further includes: receiving specification information sent by the terminal device, wherein the specification information is used to indicate the cloud-side model specification corresponding to the features of the input content; The step of performing full inference and the first X-1 incremental inferences on the prompt words includes: performing full inference and the first X-1 incremental inferences on the prompt words according to the model specifications indicated by the specification information.

16. The method according to any one of claims 10-15, characterized in that, The receiving of a prompt word from the end device includes: receiving a preprocessed prompt word from the end device; the preprocessing includes compression or encryption. The step of performing full inference and the first X-1 incremental inferences on the prompt word includes: performing the inverse operation of the preprocessing on the prompt word, and then performing full inference and the first X-1 incremental inferences.

17. A reasoning device, characterized in that, The device includes: The sending unit is used to send prompt words corresponding to the input content to the cloud server; A receiving unit is configured to receive, from the cloud server, the first X lexical units obtained by the cloud server through reasoning on the prompt word, and intermediate results; the intermediate results include one or more of the following: the first identifier ID sequence of the lexical units input by the cloud server's full reasoning, and the KV cache of the first Y lexical units among the first X lexical units obtained by the cloud server's reasoning; where X is greater than or equal to 1; and Y is greater than or equal to 1 and less than or equal to X; the full reasoning is used to obtain the first character in the answer corresponding to the prompt word. The output unit is used to output the first X words; The reasoning unit is used to perform reasoning based on the intermediate results to obtain and output the remaining words in the answer.

18. A reasoning device, characterized in that, The device includes: The receiving unit is used to receive prompts from the end device; The reasoning unit is used to perform full reasoning and the first X-1 incremental reasoning on the prompt words to obtain the first X word elements; The sending unit is used to send the first X lexical units and intermediate results to the terminal device; the intermediate results include one or more of the following: the first identifier ID sequence of the lexical units input by full inference, and the KV cache of the first Y lexical units in the first X lexical units obtained by inference; X is greater than or equal to 1; Y is greater than or equal to 1 and less than or equal to X; the full inference is used to obtain the first character in the answer corresponding to the prompt word.

19. A computing device, characterized in that, The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the operational steps of the method as described in any one of claims 1 to 16.

20. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the operational steps of the method as described in any one of claims 1 to 16.

21. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the operational steps of the method as described in any one of claims 1 to 16.

Images