Data processing method, large language model deployment method and related apparatus

Abstract

Disclosed in the present application are a data processing method, a computing device and a related apparatus. In the method, by means of deploying, in a first trusted execution environment of a first computing node, a large language model and a security operator that are run in a prefilling phase, and deploying, in a second trusted execution environment of a second computing node, a large language model and a security operator that are run in a decoding phase, data communication between the computing nodes is implemented by using a secure data channel. During inference computation, first, the decryption of a prompt and prefilling computation are completed at the first computing node to obtain an intermediate computation result corresponding to the prompt; and then, the prompt and the intermediate computation result are sent to the second computing node by means of the secure data channel, so that the second computing node performs decoding computation to obtain an inference result corresponding to the prompt. By means of the method, the potential data security risk in a data processing scheme for a PD disaggregated architecture is addressed, the security of a large language model data inference process is improved, and the privacy of users is protected.

Description

A method and related apparatus for data processing and deployment of large language models

[0001] This application claims priority to Chinese Patent Application No. 202510223723.6, filed on February 25, 2025, entitled “A method and apparatus for data processing and deployment of a large language model”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, computing device and related apparatus for a large language model. Background Technology

[0003] The use cases for large language models typically include client-side and server-side scenarios. The client-side includes a large language model application that receives user input prompts and sends them to the server. The server deploys a large language model (LLM), which performs inference calculations on the prompts, obtains the corresponding inference results, and returns these results to the client for the user to receive.

[0004] Inference computation in large language models typically includes a prefill (P) phase and a decoding (D) phase. In the prefill phase, the large language model performs parallel processing on large volumes of user input, including preprocessing, embedding layer transformation, and forward propagation, to obtain a key-value (KV) cache corresponding to the prompt word and the next lexicon. Then, in the decoding phase, the large language model calculates the prompt word, its corresponding KV cache, and the next lexicon to progressively deduce the inference result corresponding to that prompt word.

[0005] Because the pre-filling stage requires significant computational resources, and the decoding stage requires maintaining the sequence of key-value caches, thus consuming substantial memory resources, their requirements differ. Therefore, servers deploying large language models typically employ a PD (Pre-Package Execution) separation architecture, running the pre-filling stage on pre-filling compute nodes and the decoding stage tasks on decoding compute nodes.

[0006] However, the current PD-separated architecture's data processing scheme does not consider potential data security risks. For example, there is a risk of data hijacking during the process of the pre-filled computing node sending the KV cache and the next lexical term to the decoding computing node. We hope for improved solutions to enhance data security in large language model use cases to protect user data privacy. Summary of the Invention

[0007] This application provides a data processing method, computing device, and related apparatus to solve the data security problem in the data inference process of the PD separation architecture and protect users' data privacy.

[0008] Firstly, this application provides a data processing method applied to a first computing node, which includes a pre-filling layer for deploying a large language model and a first trusted execution environment for a first secure operator. The first trusted execution environment is connected to a second computing node via a secure data channel. The method may include: receiving first encrypted data from a client 100 and decrypting the first encrypted data in the first trusted execution environment to obtain a prompt word input by the user; inputting the prompt word into the pre-filling layer of the large language model, where the pre-filling layer uses the first secure operator to calculate the prompt word, obtaining an intermediate calculation result. The intermediate calculation result includes a key-value cache corresponding to the prompt word and the next lexical term; and sending the prompt word and the intermediate calculation result to the second computing node via the secure data channel, so that the second computing node obtains an inference result corresponding to the prompt word based on the prompt word and the intermediate calculation result.

[0009] In this application, a first trusted execution environment is set up in the first computing node performing the pre-filling computation task in a PD-separated architecture, and connected to a second computing node performing the decoding task through a secure data channel. The first computing node, within the first trusted execution environment, utilizes a pre-deployed large language model pre-filling layer and a first secure operator to perform pre-filling computation on the user-inputted prompt words, and sends the intermediate results to the second computing node through the secure data channel. This method ensures that the customer prompt words do not leave the secure area during the pre-filling process, achieves data security during transmission between the first and second computing nodes, and protects the user's data privacy.

[0010] In some embodiments, the client includes a pre-deployed first security toolkit that can encrypt user-input prompts.

[0011] In this application, by deploying the first security toolkit on the client, the risk of user input prompts being leaked due to Trojan implantation on the client can be avoided, thus protecting user data privacy.

[0012] In some embodiments, the first computing node may further include a first general execution environment for deploying a first general operator. The first computing node is connected to the second computing node via a conventional data channel. In this case, the method further includes: determining whether the prompt word involves user privacy data based on the prompt word; if the prompt word does not involve user privacy data, inputting the prompt word into the pre-filling layer of the large language model, whereby the pre-filling layer of the large language model uses the first general operator to calculate the prompt word and obtain an intermediate calculation result; and sending the prompt word and the intermediate calculation result to the second computing node via the conventional data channel, so that the second computing node can calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word.

[0013] In this application, a first general execution environment is set up in the first computing node performing the pre-filling computation task in the PD separation architecture, and connected to the second computing node performing the decoding task through a regular data channel. When user data does not involve user privacy, the first computing node uses the pre-filling layer of the large language model and the first general operator to perform pre-filling computation on the prompt words input by the user, and sends the intermediate results to the second computing node through the regular data channel. This method distinguishes between the user's general computation needs and secure computation needs. It enables more flexible inference computation on the user's prompt words, improves the performance of pre-filling computation without infringing on user privacy, reduces security overhead during data transmission, and achieves fine-grained data security protection.

[0014] In some embodiments, to improve the inference performance of the pre-filled layer of the large language model, the first computing node can also be connected to a storage node with a knowledge database deployed thereon. This storage node further includes a pre-deployed second security data packet. When the pre-filled layer of the large language model uses the first security operator to calculate the prompt words and obtain intermediate calculation results, the method further includes: receiving encrypted vector text sent by the storage node and decrypting the encrypted vector text in a first trusted execution environment to obtain retrieval enhancement generation data. The encrypted vector text is encrypted by the second security data packet, and the retrieval enhancement generation data is obtained from the knowledge database based on prompt word retrieval. The first computing node can input the prompt words and retrieval enhancement generation data into the pre-filled layer of the large language model, whereby the pre-filled layer uses the first security operator to calculate the prompt words and retrieval enhancement generation data to obtain intermediate calculation results.

[0015] In this application, by pre-deploying a second security data packet on the storage node, data encryption is achieved during the transmission of enhanced retrieval data. This avoids the risk of data leakage caused by Trojan implantation on the client side and protects the user's data privacy.

[0016] Secondly, this application provides a data processing method applied to a second computing node, which includes a second trusted execution environment for deploying a decoding layer of a large language model and a second secure operator. The second trusted execution environment is connected to a first computing node via a secure data channel. The method includes: receiving a prompt word and intermediate calculation results sent by the first computing node through the secure data channel. The intermediate calculation results include a key-value cache corresponding to the prompt word and a next term. The prompt word is input into the decoding layer of the large language model, where the decoding layer uses the second secure operator to calculate the prompt word and the intermediate calculation results to obtain an intermediate calculation result. The intermediate calculation result includes a key-value cache corresponding to the prompt word and a next term. The inference result is encrypted in the second trusted execution environment to obtain second encrypted data, and the second encrypted data is returned to the client.

[0017] In this application, a second trusted execution environment is set up in the second computing node performing the decoding computation task in the PD separation architecture, and connected to the first computing node performing the decoding task through a secure data channel. The second computing node receives the prompt words and intermediate computation results sent by the first computing node through the secure data channel. In the second trusted execution environment, the second computing node uses a pre-deployed large language model decoding layer and a second secure operator to decode the user-input prompt words and intermediate computation results, obtaining the inference result corresponding to the prompt words. The inference result is then encrypted and sent to the client. This method ensures that the client's prompt words do not leave the secure area during the decoding process, achieves data security during transmission between the second computing node and the client, and protects the user's data privacy.

[0018] In some embodiments, the client includes a pre-deployed security toolkit that can encrypt user-input prompts.

[0019] In this application, by deploying the first security toolkit on the client, the risk of leakage of inference results sent by the second computing node due to Trojan implantation on the client can be avoided, thus protecting the user's data privacy.

[0020] In some embodiments, the second computing node may further include a second general execution environment, which is connected to the first computing node via a conventional data channel. The second general execution environment is used to deploy a second general operator. In this case, the method further includes: receiving a prompt word and intermediate calculation results sent by the first computing node via the conventional data channel; inputting the prompt word and intermediate calculation results to the decoding layer of the large language model; and having the decoding layer of the large language model use the second general operator to calculate the prompt word and the intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0021] In this application, a second general execution environment is set up in the second computing node performing the decoding computation task in the PD separation architecture, and connected to the second computing node performing the pre-filling computation task through a regular data channel. When user data does not involve user privacy, the second computing node uses the pre-filling layer of the large language model and the second general operator to decode the prompt words and intermediate computation results, obtaining the inference result corresponding to the prompt words, and sending the inference result to the client. This method distinguishes between the user's general computation needs and secure computation needs. It enables more flexible inference computation on the intermediate computation results of the user's prompt words, improves the performance of decoding computation without involving user privacy, reduces security overhead during data transmission, and achieves fine-grained data security protection.

[0022] In some embodiments, to improve the inference performance of the decoding layer of the large language model, the second computing node can also be connected to a storage node with a knowledge database deployed thereon. This storage node further includes a pre-deployed second security data packet. When the decoding layer of the large language model uses the second security operator to perform inference calculations on the prompt words and intermediate calculation results, the method further includes: receiving encrypted vector text sent by the storage node and decrypting the encrypted vector text in a second trusted execution environment to obtain retrieval enhancement generation data. The encrypted vector text is encrypted by the second security data packet, and the retrieval enhancement generation data is obtained from the knowledge database based on prompt word retrieval. The first computing node can input the prompt words, intermediate calculation results, and retrieval enhancement generation data into the decoding layer of the large language model. The decoding layer of the large language model uses the second security operator to calculate the prompt words, intermediate calculation results, and retrieval enhancement generation data to obtain the inference result corresponding to the prompt word.

[0023] In this application, by pre-deploying a second security data packet on the storage node, data encryption is achieved during the transmission of enhanced retrieval data. This avoids the risk of data leakage caused by Trojan implantation on the client side and protects the user's data privacy.

[0024] Thirdly, embodiments of this application provide a method for deploying a large language model, which can be executed by any computing device. A computing device can use a third security toolkit to encrypt the pre-filled layer, the first security operator, the decoding layer, and the second security operator of the large language model. In practice, firstly, the computing device needs to split the large language model into a pre-filled layer and a decoding layer. Then, the pre-filled layer can recognize the first security operator. The decoding layer can also recognize the second security operator. Next, the computing device can encrypt the pre-filled layer, the first security operator, the decoding layer, and the second security operator. Finally, the computing device can send the encrypted pre-filled layer and the first security operator of the large language model to the first trusted execution environment of the first computing node, so that the first computing node can decrypt the pre-filled layer and the first security operator in the first trusted execution environment to complete the deployment of the pre-filled layer and the first security operator of the large language model. The computing device can also simultaneously send the encrypted decoding layer of the large language model and the second security operator to the second trusted execution environment of the second computing node, so that the second computing node can decrypt the decoding layer of the large language model and the second security operator in the second trusted execution environment to complete the deployment of the decoding layer of the large language model and the second security operator.

[0025] In this application, by pre-deploying the pre-filling layer of the large language model and the first security operator in the pre-filling stage of the execution phase in the first trusted execution environment of the first computing node, and pre-deploying the decoding layer of the large language model and the second security operator in the decoding stage of the execution phase in the second trusted execution environment of the second computing node, a reliable security foundation can be provided for subsequent data processing methods. This method achieves data security during the deployment of the large language model and security operators, improves data security during data inference, and protects user data privacy.

[0026] Fourthly, this application proposes a pre-filling processing apparatus. This apparatus is applied to a first computing node and connected to a second computing node via a secure data interface. The apparatus includes a first data security module and a pre-filling module. The first data security module receives first encrypted data sent by a client and decrypts the first encrypted data in a first trusted execution environment (TEX) on the first computing node to obtain a prompt word input by the user. The first TEX is used to deploy a pre-filling layer of a large language model and a first secure operator. The pre-filling module inputs the prompt word into the pre-filling layer of the large language model, where the pre-filling layer uses the first secure operator to calculate the prompt word, obtaining an intermediate calculation result. The intermediate calculation result includes a key-value cache corresponding to the prompt word and the next lexical term. The pre-filling module also sends the prompt word and the intermediate calculation result to the second computing node via the secure data interface, so that the second computing node obtains the inference result corresponding to the prompt word based on the prompt word and the intermediate calculation result.

[0027] In some embodiments, the pre-filling processing device is also connected to the second computing node via a conventional data interface. In this case, the first data security module is further configured to determine whether the prompt word involves the user's privacy data based on the prompt word. If the prompt word does not involve the user's privacy data, the pre-filling module is configured to input the prompt word into the pre-filling layer of the large language model, whereby the pre-filling layer of the large language model calculates the prompt word using a first general operator to obtain an intermediate calculation result. The first general operator is an operator deployed in the first general execution environment of the first computing node. The pre-filling module is also configured to send the prompt word and the intermediate calculation result to the second computing node via a conventional data channel, so that the second computing node can calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word.

[0028] Fifthly, this application proposes a decoding processing apparatus. The decoding processing apparatus is applied to a second computing node and connected to a first computing node via a secure data interface. The apparatus includes a decoding module and a second data security module. The decoding module is used to receive prompt words and intermediate calculation results sent by the first computing node through the secure data interface within a second trusted execution environment on the second computing node. The intermediate calculation results include a key-value cache corresponding to the prompt word and the next term. The second trusted execution environment is used to deploy the decoding layer of a large language model and a second secure operator. The decoding module is also used to input the prompt word and intermediate calculation results into the decoding layer of the large language model, whereby the decoding layer uses the second secure operator to calculate the prompt word and the intermediate calculation results to obtain the inference result corresponding to the prompt word. The second data security module is used to encrypt the inference result in the second trusted execution environment to obtain second encrypted data and return the second encrypted data to the client.

[0029] In some embodiments, the decoding processing device is also connected to the first computing node via a conventional data interface. In this case, the decoding module is used to receive the prompt word and intermediate calculation results sent by the first computing node through the conventional data interface in the second general execution environment of the second computing node. The second general execution environment includes a second general operator. The decoding module is also used to input the prompt word and intermediate calculation results into the decoding layer of the large language model, whereby the decoding layer of the large language model uses the second general operator to calculate the prompt word and the intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0030] Sixthly, embodiments of this application provide a data processor system, which includes a first computing node and a second computing node. The first computing node and the second computing node are interconnected through a secure data channel. The first computing node includes a first trusted execution environment, which pre-deploys a pre-filling layer of a large language model and a first security operator. The second computing node includes a second trusted execution environment, which pre-deploys a decoding layer of a large language model and a second security operator.

[0031] The first computing node receives first encrypted data sent by the client and decrypts it in a first trusted execution environment to obtain the prompt word input by the user. The first computing node also inputs the prompt word into the pre-filling layer of the large language model, where the pre-filling layer uses a first secure operator to calculate the prompt word, obtaining an intermediate calculation result. This intermediate calculation result includes the key-value cache corresponding to the prompt word and the next lexical term. The prompt word and the intermediate calculation result are then sent to the second computing node through a secure data channel. The second computing node inputs the prompt word and the intermediate calculation result sent by the first computing node into the decoding layer of the large language model. The decoding layer uses a second secure operator to calculate the prompt word and the intermediate calculation result, obtaining the inference result corresponding to the prompt word. The second computing node further encrypts the inference result in a second trusted execution environment to obtain second encrypted data and returns the second encrypted data to the client.

[0032] In some embodiments, the first computing node further includes a first general execution environment. The second computing node includes a second general execution environment, and the first general execution environment is connected to the second general execution environment via a conventional data channel. The first general execution environment is used to deploy a first general operator, and the second general execution environment is used to deploy a second general operator.

[0033] In this scenario, the data processing system can process the prompt words according to whether they involve the user's privacy data. The first computing node is also used to determine whether the prompt word involves the user's privacy data. If the prompt word does not involve the user's privacy data, it is input into the pre-filling layer of the large language model. The pre-filling layer of the large language model uses the first ordinary operator to calculate the prompt word and obtain an intermediate calculation result. The first computing node can send the prompt word and the intermediate calculation result to the second computing node through a regular data channel. The second computing node is also used to input the prompt word and the intermediate calculation result into the decoding layer of the large language model. The decoding layer of the large language model uses the second ordinary operator to calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word.

[0034] In a seventh aspect, this application proposes a computer-readable storage medium whose computer program instructions, when executed by a first computing node, enable the first computing node to perform the data processing method described above.

[0035] Eighthly, this application proposes a computer program product containing instructions that, when executed by a first computing node, cause the first computing node to perform the aforementioned data processing method.

[0036] Ninthly, this application proposes a computer-readable storage medium whose computer program instructions, when executed by a first computing node, enable the first computing node to perform the data processing method described above.

[0037] In a tenth aspect, this application proposes a computer program product containing instructions that, when executed by a first computing node, cause the first computing node to perform the aforementioned data processing method.

[0038] It should be understood that the beneficial effects of the technical solutions and corresponding possible implementations of the fourth, fifth, sixth, seventh, eighth, ninth and tenth aspects of the embodiments of this application can be referred to the above-described technical effects of the first and second aspects, and will not be repeated here. Attached Figure Description

[0039] Figure 1 is a schematic diagram of a scenario for protecting large model data using a trusted execution environment;

[0040] Figure 2 is a schematic diagram of the data processing scheme using the PD separation architecture;

[0041] Figure 3 is a schematic diagram of an interactive scenario of a data processing method disclosed in this application;

[0042] Figure 4 is a schematic diagram of a fully encrypted inference architecture implemented using a first processor and a second processor;

[0043] Figure 5 is a schematic diagram of the fine-grained cryptographic inference architecture implemented using the first processor and the second processor;

[0044] Figure 6 is a flowchart illustrating a data processing method proposed in this application;

[0045] Figure 7 is a flowchart illustrating another data processing method proposed in this application;

[0046] Figure 8 is a flowchart illustrating a method for deploying a large language model proposed in this application;

[0047] Figure 9 is a schematic diagram of a pre-filling processing device proposed in this application;

[0048] Figure 10 is a schematic diagram of the system architecture of a data processor system proposed in this application.

[0049] Specific implementation method

[0050] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0051] To facilitate understanding of the technical solution of this application, the relevant terms used in this document are explained below.

[0052] A Trusted Execution Environment (TEE) is a secure, isolated environment within a computer process or system, protected in terms of confidentiality and integrity. The TEE runs in an independent environment and in parallel with the operating system. The processor ensures the confidentiality and integrity of code and data within the TEE. Trusted applications running in the TEE can access the full functionality of the device's main processor and memory, while hardware isolation protects these components from user-installed applications running on the main operating system. In contrast to a Trusted Execution Environment, other execution environments within a computer process or system can be referred to as ordinary executable environments.

[0053] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. Their core idea is to learn patterns and structures of natural language through large-scale unsupervised training, thus mimicking the human language cognition and generation process to some extent.

[0054] In the field of artificial intelligence (AI), a prompt is the text or instruction provided as input to a large language model to guide it in generating a specific output. In practical applications, a prompt can be a short question, a complete paragraph, or a set of instructions, depending on the user's needs and the context. Prompts describe the information, answer, or text that the user wants to obtain from the model. When generating text, the model attempts to understand the prompts and generate a corresponding response based on its understanding.

[0055] The use cases for large language models typically include client-side and server-side scenarios. The client-side includes the large language model application, which receives user-input prompts and sends them to the server. The server-side includes the large language model, which uses its operators to perform inference calculations on the prompts, obtains the corresponding inference results, and returns these results to the client for the user to receive.

[0056] To address data protection in traditional large language model inference scenarios, existing technologies propose a data processing method. For example, Figure 1 illustrates a scenario where a trusted execution environment (TEU) is used to protect large model data. As shown in Figure 1, a traditional large language model architecture typically includes a client and a server. The client includes a central processing unit (CPU) on which the large model application is deployed. The server includes a CPU and a neural network processor. The CPU of the server includes an LLM inference service interface, through which data communication between the client and the server can be achieved. The neural network processor includes a trusted execution environment (TEU) used to deploy the large language model. Secure operators are pre-deployed within the large language model.

[0057] Next, the processing flow of input data in the reasoning scenario of a large language model will be introduced in the following content, as shown by the solid arrow in Figure 1.

[0058] First, the client's central processing unit can encrypt the plaintext prompts (hereinafter referred to as plaintext input) entered by the user, and send the encrypted ciphertext input to the server's central processing unit through the LLM inference service interface.

[0059] Then, after receiving the ciphertext input, the server's central processing unit can send the ciphertext input to the neural network processor, which will decrypt it in a trusted execution environment to obtain the plaintext input.

[0060] Finally, the neural network processor can input the plaintext into the large language model, which then uses secure operators to continue reasoning and computation to obtain the reasoning result corresponding to the plaintext input.

[0061] Next, the processing flow of output data in the reasoning scenario of a large language model will be introduced in the following content, as shown by the dashed arrow in Figure 1.

[0062] First, the neural network processor can encrypt the plaintext inference results (hereinafter referred to as output plaintext) output by the large language model in a trusted execution environment to obtain the output ciphertext corresponding to the input plaintext.

[0063] Then, the neural network processor can send the output ciphertext to the server's central processing unit, which then sends the output ciphertext to the client's central processing unit through the LLM inference service interface.

[0064] Finally, the client's central processing unit decrypts the ciphertext to obtain the plaintext, and sends the plaintext to the large model application for the user to receive.

[0065] In practice, the inference computation of a large language model typically includes two stages: prefill (P) and decoding (D).

[0066] During the pre-filling stage, the large language model performs a series of data processing steps on the user-input prompts, including preprocessing, embedding layer transformation, and forward propagation, to obtain a key-value (KV) cache corresponding to the prompts. Since the pre-filling stage involves parallel processing of large amounts of data, it typically requires significant computational power and resources.

[0067] During the decoding phase, the large language model can generate each word sequentially by calculating the aforementioned key-value cache and the context information associated with it. This word is then used as input for the next generation, progressively deriving the inference result corresponding to the prompt word. Since the inference task in the decoding phase is sequential, the inference speed is relatively slow, requiring less computational resources compared to the pre-filling phase. However, maintaining the sequence of the key-value cache requires significant memory resources during the decoding phase.

[0068] Because the pre-filling stage and the decoding stage have different requirements, the traditional large language model architecture shown in Figure 1 (which deploys the pre-filling stage and the decoding stage on the same physical computing card or processor) often faces the problem of uneven resource utilization.

[0069] To address this characteristic, existing technologies have optimized the traditional large language model architecture. The optimized large language model server typically employs a PD (Pre-Package and Decoding) separation architecture, running the pre-filling and decoding tasks on different physical computing cards or processors.

[0070] For example, Figure 2 is a schematic diagram of a data processing scheme using a PD separation architecture. As shown on the left side of Figure 2, the PD separation architecture can include multiple pre-filled instances and multiple decoding instances, such as pre-filled computing node 1 and decoding computing node 1. The specific implementation of pre-filled computing node 1 and decoding computing node 1 can be mutually isolated physical computing cards, or any two processors integrated into the same computing device, or deployed together on a single computing device at the granularity of virtual machines or containers.

[0071] The pre-filled compute node 1 includes a local scheduler 1, a length predictor, a scheduler, a pre-filled LLM main engine, and a KV cache 1.

[0072] Local scheduler 1 is used to receive multiple raw requests input by the user, sort the multiple raw requests according to the pre-set scheduling strategy, and input the multiple raw requests one by one into the length predictor and the pre-filled LLM main engine in sequence.

[0073] The length predictor pre-deploys a length prediction model, which predicts the length of the inference result corresponding to each original request and determines the resource usage of the original request during the decoding stage based on the length of the inference result. Then, the length predictor can send the predicted result of the original request to the scheduler.

[0074] The pre-filled LLM main engine pre-deploys a large language model, which is used to transform a raw request into easily identifiable prompt words and the corresponding decoding request. The large language model is also used to segment the prompt words into multiple data blocks and pre-fill each data block.

[0075] In practice, the pre-filled LLM main engine's large language model can perform a series of calculations on each data block, including preprocessing, embedding layer transformation, and forward propagation, to generate the intermediate calculation results corresponding to the aforementioned prompt words. These intermediate calculation results can include a key-value cache and the next token, which can be a word, character, or subword.

[0076] The scheduler maintains a scheduling queue for multiple decoding requests. When scheduling a job for a decoding request, the scheduler can determine which decoding instance should handle the request from among multiple decoding instances based on the prediction result sent by the length predictor.

[0077] The scheduler is also used to receive intermediate results and status information returned by the pre-populated LLM main engine, and to make further scheduling decisions based on the intermediate results and status information, such as adjusting task allocation and increasing resources.

[0078] KV cache 1 includes a populated key cache and a populated value cache, which are used to store the KV cache generated in the pre-population stage.

[0079] Taking the scheduling program determining that decoding computing node 1 will handle the above decoding request as an example, pre-filled computing node 1 can send the decoding request, the next word corresponding to the prompt word, and the KV buffer to decoding computing node 1.

[0080] Decoding compute node 1 includes a local scheduler 2, a decoding LLM main engine, and a KV cache 2.

[0081] Local scheduler 2 is used to receive multiple decoding requests sent by pre-filled computing node 1, and send the multiple decoding requests to the decoding LLM main engine according to the scheduling policy.

[0082] The main engine for decoding LLM also pre-deploys a large language model, which is typically decoded in parallel or in batches.

[0083] The large language model in the decoding LLM main engine can generate new lexicals for each request in an autoregressive manner based on multiple decoding requests sent by the local scheduler 2, along with the corresponding next lexical and KV cache, to obtain the inference result for each request. Simultaneously, during the generation of these inference results, the large language model in the decoding LLM main engine can also update the KV cache based on contextual information.

[0084] KV cache 2 includes a populated key cache, a populated value cache, and a decoded KV cache.

[0085] The populated key cache $ and populated value cache $ are used to store the pre-populated key-value cache sent by compute node 1 and the updated key-value cache.

[0086] The KV cache after decoding is used to store historical information about intermediate calculation results and inference results during the decoding process.

[0087] However, the current PD-separated architecture's data processing solution does not consider potential data security risks. These include the risk of malware injection on the client and server sides, and the risk of data hijacking during the process of pre-filling the compute node sending the KV cache and the next lexical term to the decoding compute node. We hope for improved solutions to enhance data security in large language model usage scenarios to protect user data privacy.

[0088] To enhance the security of large language model data inference, this application discloses a data processing method, computing device, and related apparatus. This method deploys the large language model and secure operators running the pre-filling phase in a first trusted execution environment (TEX) of a first computing node, and the large language model and secure operators running the decoding phase in a second trusted execution environment of a second computing node. Data communication between these computing nodes is achieved using a secure data channel. During inference computation, the first computing node first decrypts and pre-fills the prompt word to obtain the intermediate computation result corresponding to that prompt word. Then, the prompt word and the intermediate computation result are sent to the second computing node via the secure data channel for decoding computation to obtain the inference result corresponding to the prompt word. This method addresses the potential data security risks of data processing schemes in a PD (Programming Processing) separation architecture, enhances the security of the large language model data inference process, and protects user privacy.

[0089] For example, Figure 3 is a schematic diagram of an interactive scenario of a data processing method disclosed in this application, as shown in Figure 3. The use case of the large language model may include a client 100 and a computing device 200.

[0090] Client 100 includes a large model application 110 and a security toolkit 111 (first security data packet) deployed within the large model application. The large model application 110 receives prompts input by user 10 and encrypts the prompts using the security toolkit 111 to obtain ciphertext input corresponding to the prompts. The large model application 110 can then send the ciphertext input to computing device 200.

[0091] The client 100 is also used to receive the ciphertext output returned by the computing device 200, and to decrypt the ciphertext output using the secure data packet 111 to obtain the reasoning result corresponding to the above prompt words, so that the user 10 can view it.

[0092] It is worth noting that client 100 can be a standalone computing device deployed independently, or it can be a container or virtual machine deployed on a computing device. In other words, this application does not limit the deployment method of client 100.

[0093] The computing device 200 includes a first computing node 210 and a second computing node 220, and a secure data channel 230 for connecting the first computing node 210 and the second computing node 220. The secure data channel 230 may include a data channel implemented using confidential computing technology, specifically a data channel implemented using a Peer-to-Peer Interconnect for Confidential Computing (PCIPC) architecture.

[0094] It is worth noting that the above-described centralized deployment, such as integrating the first computing node 210 and the second computing node 220 onto the computing device 200, is intended to facilitate understanding of this application. In actual implementation, this application can also adopt a distributed deployment approach, placing multiple computing nodes identical to the first computing node 210 in one computing cluster, and multiple computing nodes identical to the second computing node 220 in another computing cluster, or deploying all computing nodes in one computing cluster. In other words, this application does not limit the deployment method of the first computing node 210 and the second computing node 220.

[0095] By way of example and not limitation, the first computing node 210 may include a container or virtual machine running on a computing device, or a processor with a general-purpose architecture that implements one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), such as the Kunpeng processor using a reduced instruction set architecture. In other words, this application does not limit the implementation of the first computing node 210.

[0096] By way of example and not limitation, the second processor 210 may include a container or virtual machine on a computing device, or it may include an artificial intelligence algorithm processor that is one or more implementations of a Compute Architecture for Neural Networks (CANN) Deep Learning Processing Unit (DPU), Tensor Processing Unit (TPU), or Neural Network Processing Unit (NPU). It may also be a container or virtual machine on a computing device. For example, an Ascend processor employing a heterogeneous computing architecture. In other words, this application does not limit the implementation of the second computing node 220.

[0097] A first trusted execution environment is deployed on the first computing node 210. The first trusted execution environment includes a pre-filling layer 211 for a large language model and at least one pre-deployed security operator, such as security operator 212 (first security operator). The security operator 212 is used to pre-fill privacy data (such as financial information such as banks and stocks) related to the aforementioned prompt words.

[0098] The first computing node 210 is used to receive the ciphertext input (first encrypted data) sent by the client 100, and decrypt the ciphertext input in the first trusted execution environment to obtain the decrypted prompt word.

[0099] The first computing node 210 can input the above prompt words into the pre-filling layer 211 of the large language model. The pre-filling layer 211 of the large language model can use the safety operator 212 to perform a series of data processing such as preprocessing, embedding layer transformation and forward propagation on the prompt words to obtain intermediate calculation results, as shown in Figure 3. The above intermediate calculation results include the KV cache and the next word element corresponding to the prompt word.

[0100] The first computing node 210 can send the prompt word and intermediate calculation results to the second computing node 220 through the secure data channel 230.

[0101] A second trusted execution environment is deployed on the second computing node 220. The second trusted execution environment includes a decoding layer 221 for a large language model and at least one pre-deployed security operator, such as security operator 222 (the second security operator). The security operator 222 is used to decode and compute the privacy data related to the aforementioned prompt words.

[0102] In some possible implementations, the second computing node 220 can also be used to receive ciphertext input sent by the client 100 and decrypt the ciphertext input sent by the client 100 in the second trusted execution environment to obtain the decrypted prompt word (not shown in the figure).

[0103] The second computing node 220 is also used to receive the intermediate calculation results corresponding to the prompt words sent by the first computing node 210.

[0104] The second computing node 220 can input the above prompt words and the intermediate calculation results corresponding to the prompt words into the decoding layer 221 of the large language model. The pre-filling layer 221 of the large language model uses the security operator 222 to perform inference calculation based on the above prompt words and the intermediate calculation results corresponding to the prompt words to obtain the inference result corresponding to the above prompt words.

[0105] It is worth noting that, before deploying the pre-filling layer and the decoding layer of the large language model, this application can utilize encryption tools to encrypt and protect the pre-filling layer and the decoding layer of the large language model.

[0106] The large language model can be implemented using a very large language inference framework (also known as vLLM) or the Ollam inference framework. In other words, this application does not limit the specific implementation method of the large language model.

[0107] The first computing node 210 can encrypt the above inference results in the second trusted execution environment to obtain the encrypted ciphertext output, and return the ciphertext output to the client 100.

[0108] To address the general computing needs of user 10, in some possible implementations, at least one general operator, such as general operator 213, can be deployed in the first trusted execution environment, and at least one general operator, such as general operator 223, can be deployed in the second trusted execution environment. The aforementioned general operators 213 and 223 can be used to perform calculations on user 10's non-privacy data (such as food delivery data, shopping information, etc.).

[0109] As an example rather than a limitation, the aforementioned first and second trusted execution environments can be execution environments implemented using traditional CPU confidential computing technologies, such as Intel TDX, AMD SEV, and ARM Trustzone architectures, or they can be execution environments implemented using heterogeneous computing power confidential computing schemes, such as Nvidia's AI confidential computing architecture. In other words, this application does not limit the implementation method of the first and second trusted execution environments.

[0110] Since the knowledge in large language models is limited by the training data, it cannot be updated in real time. Therefore, in some possible implementations, vector databases can be used to improve the accuracy and relevance of the inference results of large language models.

[0111] Please refer to Figure 3. Illustratively, a use case for a large language model may include storage node 300. Storage node 300 includes a vector database 310 and a security toolkit 320 (second security data package). The vector database 310 includes pre-stored unstructured vector data (such as images, audio, and text).

[0112] Next, the following content will introduce the use cases of large language models using vector databases for reasoning.

[0113] First, the client 100 can generate a knowledge vector based on the prompt words input by the user, encrypt the knowledge vector using the security toolkit 111 to obtain an encrypted vector, and send the encrypted vector to the storage node 300, as shown by the dashed arrow in Figure 3. This will not be elaborated further below.

[0114] Then, the storage node 300 can use the security toolkit 320 to decrypt the encrypted vector to obtain the decrypted knowledge vector, and retrieve the retrieval augmented generation (RAG) data corresponding to the prompt words input by the user 10 from the knowledge database 310 based on the decrypted knowledge vector.

[0115] Then, the storage node 300 can use the security toolkit 320 to encrypt the RAG data to obtain encrypted vector text, and send the encrypted vector text to the computing device 200.

[0116] Then, after receiving the encrypted vector text, the computing device 200 can send the encrypted vector text to the processor 210 and the processor 220 respectively.

[0117] Finally, the processor 210 can decrypt the encrypted vector text to obtain the RAG data corresponding to the knowledge vector. The processor 210 can then enhance the prompts input by the user based on the aforementioned RAG data, and input the enhanced prompts into the pre-filling layer of the large language model so that the pre-filling layer of the large language model can output intermediate calculation results.

[0118] Understandably, processor 220 can also decrypt encrypted vector text to obtain RAG data corresponding to the knowledge vector. Processor 220 can then enhance the user-input prompts and next words based on the aforementioned RAG data, and input the enhanced prompts and next words to the decoding layer of the large language model. The decoding layer of the large language model can then output the inference result corresponding to the aforementioned prompts based on the enhanced prompts, next words, and the KV cache.

[0119] It is worth noting that the vector database deployed using a single storage node shown in Figure 3 is intended to facilitate understanding of this application. In actual implementation, the vector database can also be deployed on a storage cluster or in the cloud. In other words, this application does not limit the actual deployment method of the vector database.

[0120] In some possible implementations, to enhance the security of user data during the data processing of large language models, it is also possible to disregard the distinction between ordinary operators and security operators, deploying the computational tasks of the pre-filling stage entirely in a first trusted execution environment, and deploying the computational tasks of the decoding stage entirely in a second executable environment.

[0121] For example, Figure 4 is a schematic diagram of a fully encrypted inference architecture implemented using a first processor and a second processor. As shown in Figure 4, the first trusted execution environment of the first computing node 210 includes a pre-filling layer 211 for a large language model, a security operator 212, a KV cache 1, and processor resources for performing pre-filling tasks. Schematic, the aforementioned processor resources may include computing cores 1 to 8 of the processor 210.

[0122] The second trusted execution environment of the second computing node 220 includes a large language model decoding layer 221, a security operator 222, a KV cache 2, and processor resources for performing decoding tasks. Illustratively, the aforementioned processor resources may include computing cores 1 to 8 of the processor 220.

[0123] The first trusted execution environment and the second trusted execution environment communicate with each other through the secure data channel 230.

[0124] When performing data processing during the pre-filling phase, the first computing node 210 can first receive the first encrypted data sent by the client 100, and decrypt the first encrypted data in the first trusted execution environment to obtain the prompt words input by the user.

[0125] Then, the first computing node 210 can input the above prompt words into the pre-filling layer 211 of the large language model. The pre-filling layer 211 of the large language model performs word segmentation on the above prompt words, divides the prompt words into multiple data blocks, and pre-fills each data block.

[0126] Then, the pre-filling layer 211 of the large language model can use the safety operator 212 to perform a series of calculations such as preprocessing, embedding layer transformation, and forward propagation on each data block to generate the intermediate calculation results corresponding to the prompt word. The intermediate calculation results can include the KV cache corresponding to the prompt word and the next word element, which can be a word, a character, or a subword.

[0127] When the pre-filling layer 211 of the large language model performs the computational tasks of the pre-filling stage, the computing power can be provided by the computing cores 1 to 8 of the first processor.

[0128] Finally, the first computing node 210 can send the aforementioned prompt words and intermediate calculation results to the second trusted execution environment of the second computing node 220 through the secure data channel 230, so that the second processor can perform the calculation task in the decoding stage.

[0129] When performing data processing during the decoding phase, firstly, the second computing node 220 can input the prompt words sent by the first processor and the intermediate calculation results into the decoding layer 221 of the large language model in the second trusted execution environment.

[0130] Then, the decoding layer 221 of the large language model can use the security operator 222 to gradually generate new lexical units corresponding to the above prompt words in an autoregressive manner, so as to obtain the inference result corresponding to the above prompt words.

[0131] Schematic illustration: when the prompt word sent by the first computing node 210 is "I love to eat", the next lexical word corresponding to this prompt word is "fish". The decoding layer 221 of the large language model can gradually generate the inference result "I love to eat shredded pork with garlic sauce" corresponding to the prompt word "I love to eat" based on the above content and the KV cache. At the same time, in the process of generating the above inference result, the decoding layer 221 of the large language model can also update the KV cache based on the context information.

[0132] Finally, the second processor 210 can encrypt the above reasoning result to obtain the second encrypted data, and send the second encrypted data to the client 100 so that the client can decrypt it and display the reasoning result "I love to eat shredded pork with garlic sauce" to the client 100.

[0133] It is worth noting that the above-described use of the first computing node 210 and the second computing node 220 to implement the reasoning task of a single prompt word is intended to facilitate understanding of this application. In actual implementation, the first computing node 210 and the second computing node 220 may also include a scheduler and a hardware accelerator (not shown in the figure) so that the first computing node 210 and the second computing node 220 can process multiple prompt words sent by one or more clients in parallel or in batches. This application will not elaborate on this further.

[0134] To achieve finer-grained data security protection during the data processing of large language models, this application can deploy the common operator 213 shown in Figure 3 in the general executable environment of the first computing node 210, and deploy the common operator 223 in the general executable environment of the second computing node 220, so as to realize the inference calculation of prompt words without involving user privacy data.

[0135] For example, Figure 5 is a schematic diagram of a fine-grained cryptographic inference architecture implemented using a first processor and a second processor. As shown in Figure 5, this inference architecture differs from that in Figure 4 in the following ways.

[0136] 1) The first computing node 210 includes a first general-purpose executable environment, which includes a pre-filling layer 211 for a large language model, a general operator 213, and processor resources for performing user non-privacy data pre-filling tasks. Schematic, the processor resources may include computing cores 1 to 6 of the processor 210.

[0137] 2) The second processor 210 includes a second general-purpose executable environment, which includes a decoding layer 221 for a large language model, general operators 223, and processor resources for performing user non-privacy data decoding tasks. Illustratively, the processor resources may include computing cores 1 to 6 of the processor 220.

[0138] 3) The first and second common executable environments are connected using traditional data channels, such as Peripheral Component Interconnect Express (PCIe) or UltraPath Interconnect (UPI).

[0139] 4) The first trusted execution environment includes computing core 7 and computing core 8 of the first computing node 210.

[0140] 5) The second trusted execution environment includes computing core 7 and computing core 8 of the second computing node 220.

[0141] The first trusted execution environment and the second executable environment still communicate using the secure data channel 230.

[0142] When performing data processing during the pre-filling phase, the first computing node 210 can first receive the first encrypted data sent by the client 100, and decrypt the first encrypted data in the first trusted execution environment to obtain the prompt words input by the user.

[0143] Then, the first computing node 210 can judge the above prompt words to determine whether the prompt words involve private data.

[0144] If the prompt does not involve the user's privacy data, the first computing node 210 can input the prompt into the pre-filling layer 211 of the large language model, and the pre-filling layer of the large language model can use the ordinary operator 213 to perform calculations to obtain intermediate calculation results.

[0145] Then, the first computing node 210 can send the aforementioned prompt words that do not involve user privacy data, as well as the intermediate calculation results, to the second general-purpose executable environment of the second processor, so that the second processor can perform the decoding task corresponding to the prompt words.

[0146] When the pre-filling layer 211 of the large language model performs the computational task of the non-privacy data pre-filling stage, the computing power can be provided by the computing cores 1 to 6 of the first processor.

[0147] If the prompt does not involve the user's private data, the first computing node 210 can input the prompt into the pre-filling layer 211 of the large language model, and the pre-filling layer of the large language model can use the security operator 212 to perform calculations to obtain intermediate calculation results.

[0148] Then, the first computing node 210 can send the aforementioned prompt word involving user privacy data, as well as the intermediate calculation results, to the second trusted execution environment of the second processor, so that the second processor can execute the decoding task corresponding to the prompt word.

[0149] When the pre-filling layer 211 of the large language model performs the computational task of the privacy data pre-filling stage, the computing power can be provided by the computing cores 7 and 8 of the first processor.

[0150] When processing data during the pre-filling stage, the second computing node 220 can also process the above prompt words in the same way. If the prompt words do not involve privacy data, they are processed by the ordinary operator 223 in the second ordinary executable environment. If they do involve privacy data, they are calculated by the security operator 222 in the second trusted execution environment to obtain the second encrypted data corresponding to the prompt words.

[0151] After the reasoning calculation is completed, the second computing node 220 can transmit the second encrypted data back to the client so that the client can use the encryption toolkit 111 to decrypt the second encrypted data and obtain the reasoning result corresponding to the prompt word.

[0152] In some possible implementations, this application may also utilize a large model application 110 in the client 100 to determine whether the prompt words entered by the user involve privacy data.

[0153] If the prompt word involves privacy data, the client 100 can use the security toolkit 111 to send the first encrypted data corresponding to the prompt word to the first computing node 210, so that the first computing node 210 and the second processor can perform secure inference calculations on the prompt word in their respective trusted execution environments. The secure inference process for the prompt word will not be described in detail here.

[0154] If the prompt does not involve private data, the client 100 can send the prompt to the first computing node 210 in plaintext, as shown by the dotted line in Figure 5.

[0155] Then, the first computing node 210 can input the prompt word into the pre-filling layer 211 of the large language model of the first ordinary execution environment, and the pre-filling layer 211 of the large language model can use the ordinary operator 213 to calculate the prompt word and obtain the intermediate calculation result.

[0156] Then, the first compute node 210 can send the prompt word and the intermediate calculation results to the second normal execution environment of the second compute node 220 via the PCIe bus.

[0157] Then, the second computing node 220 can input the prompt word and the intermediate calculation results to the decoding layer 221 of the large language model. The decoding layer 221 of the large language model uses the ordinary operator 223 to calculate the prompt word and the intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0158] Finally, the second computing node 220 can send the plaintext inference result to the client 100 so that the client 100 can display it to the user 10.

[0159] Firstly, based on the content described above, a data processing method provided by an embodiment of this application will be introduced. It is understood that this method is proposed based on the content described above, and some or all of the content of this method can be found in the description above.

[0160] For example, Figure 6 is a flowchart illustrating a data processing method proposed in this application. As shown in Figure 6, data processing of a large language model can be implemented through steps S610 to S630. It is understood that this method can be executed by the first computing node 210 shown in Figure 3.

[0161] S610: Receives the first encrypted data sent by the client and decrypts the first encrypted data in the first trusted execution environment to obtain the prompt word.

[0162] As mentioned above, the client 100 includes a large model application, in which a first security toolkit 111 is pre-deployed. The security toolkit 111 can be used to encrypt the prompt words entered by the user to obtain the first encrypted data.

[0163] The first computing node 210 includes a pre-filled layer for deploying a large language model and a first trusted execution environment for deploying a first secure operator. The first trusted execution environment is connected to the second computing node 220 via a secure data channel 230.

[0164] The first computing node 210 can receive the first encrypted data sent by the client 100, and decrypt the first encrypted data in the first trusted execution environment to obtain the prompt words input by the user 10.

[0165] For illustrative purposes only, and not as a limitation, when encrypting user-input prompts, Security Toolkit 111 can employ symmetric encryption algorithms such as DEA (Data Encryption Algorithm), AES (Advanced Encryption Standard), and TDEA (Triple Data Encryption Algorithm), or asymmetric encryption algorithms such as ECC (Error Correcting Code) and MH (Merkle-Hellman Knapsack Algorithm) to obtain the first encrypted data, depending on actual needs. It is understood that when decrypting the first encrypted data in the first trusted execution environment, the same key and algorithm used for encryption must be used to recover the user-input prompts. The algorithms used for data encryption and decryption will not be elaborated further below.

[0166] S620: Intermediate calculation results are obtained by using the first safety operator to calculate the prompt words through the pre-filled layer of the large language model.

[0167] As mentioned earlier, the first trusted execution environment pre-deploys a pre-filled layer of the large language model and the first security operator.

[0168] The first computing node 210 can input the prompt words entered by the user into the pre-filling layer 211 of the large language model. The pre-filling layer 211 of the large language model uses the first security operator 212 to calculate the prompt words and obtain intermediate calculation results. The intermediate calculation results include the key-value cache corresponding to the prompt words and the next word.

[0169] The first computing node 210 may also include a first general execution environment for deploying general operators 213. The first general execution environment is connected to the second computing node 220 via a conventional data channel 240.

[0170] In some possible implementations, the first computing node 210 can process the prompt word in two ways, depending on whether the prompt word involves the user's privacy data.

[0171] If the prompt word involves private data, the first computing node 210 can use the first security operator 212 to calculate the prompt word through the pre-filled layer 211 of the large language model to obtain intermediate calculation results.

[0172] If the prompt does not involve privacy data, the first computing node 210 can input the prompt into the pre-filling layer 211 of the large language model, and the pre-filling layer 211 of the large language model can use the first ordinary operator 213 to calculate the prompt and obtain the intermediate calculation result.

[0173] As one possible implementation, the first computing node can also be connected to the storage node 300. The storage node 300 310 pre-deploys a vector database and a second security data packet 320.

[0174] The first computing node 210 can receive encrypted vector text sent by the storage node 300 and decrypt the encrypted vector text in the first trusted execution environment to obtain retrieval enhancement generation data. The encrypted vector text is encrypted by the second secure data packet 320, and the aforementioned retrieval enhancement generation data is determined by the knowledge database 312 based on prompts entered by the user.

[0175] The first computing node 210 can input the prompt words and search enhancement generation data into the pre-filling layer 211 of the large language model. The pre-filling layer 211 of the large language model uses the first security operator 212 to calculate the prompt words and search enhancement generation data to obtain intermediate calculation results.

[0176] It is understandable that if the prompt words input by user 10 do not involve privacy data, the first computing node 210 can also input the prompt words and search enhancement generation data into the pre-filling layer 211 of the large language model, and the pre-filling layer 211 of the large language model can use the first ordinary operator 213 to calculate the above prompt words and search enhancement generation data to obtain intermediate calculation results.

[0177] S630: Send the prompt words and intermediate calculation results to the second computing node through a secure data channel so that the second computing node can obtain the inference results.

[0178] As previously mentioned, the first trusted execution environment can be connected to the second computing node 220 via a secure data channel.

[0179] The first computing node 210 can send the aforementioned prompt word and intermediate calculation results to the second computing node 220 through a secure data channel, so that the second computing node 220 can obtain the inference result corresponding to the prompt word based on the aforementioned prompt word and the intermediate calculation results.

[0180] The first general execution environment of the first computing node 210 can also be connected to the second computing node 220 through a regular data channel.

[0181] When the first computing node 210 includes the first normal execution environment.

[0182] In some possible implementations, if the prompt word entered by user 10 does not involve privacy data, the first computing node 210 can send the prompt word and intermediate calculation results to the second computing node 220 through a regular data channel, so that the second computing node 220 can perform calculations on the prompt word and the intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0183] In a PD-separated architecture, a first trusted execution environment is set up in the first computing node performing the pre-filling computation task, which is connected to a second computing node performing the decoding task via a secure data channel. The first computing node, within the first trusted execution environment, utilizes a pre-deployed pre-filling layer of a large language model and a first secure operator to perform pre-filling computation on the user-inputted prompt words, and sends the intermediate results to the second computing node via the secure data channel. This method ensures that the customer prompt words do not leave the secure area during the pre-filling process, achieves data security during transmission between the first and second computing nodes, and protects the user's data privacy.

[0184] Secondly, based on the content described above, another data processing method provided in the embodiments of this application will be introduced. It is understood that this method is proposed based on the content described above, and some or all of the content of this method can be found in the description above.

[0185] For example, Figure 7 is a flowchart illustrating another data processing method proposed in this application. As shown in Figure 7, data processing of a large language model can be implemented through steps S710 to S730. It is understood that this method can be executed by the second computing node 220 shown in Figure 3.

[0186] S710: Receives prompts and intermediate calculation results sent by the first computing node through a secure data channel.

[0187] As previously described, the second computing node 220 includes a second trusted execution environment for deploying a decoding layer for a large language model and a second secure operator. The second trusted execution environment can be connected to the first computing node via a secure data channel 230.

[0188] The second computing node 220 can receive the prompt word and intermediate calculation results sent by the first computing node 210 through the secure data channel 230. The intermediate calculation results include the key-value cache corresponding to the prompt word and the next word element.

[0189] The second computing node 220 may also include a second general execution environment for deploying the second general operator 223. The second general execution environment is connected to the first computing node 210 via a conventional data channel 240.

[0190] In some possible implementations, if the prompt entered by user 10 does not involve privacy data, the second computing node 220 can receive the prompt and intermediate calculation results sent by the first computing node 210 through the regular data channel 240.

[0191] S720: The second security operator is used in the decoding layer of the large language model to calculate the inference result corresponding to the prompt word and the intermediate calculation result.

[0192] The second computing node 220 can input the prompt word and intermediate calculation results to the decoding layer 221 of the large language model. The decoding layer 221 of the large language model uses the second security operator 222 to calculate the prompt word and intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0193] When the first computing node 210 includes the first normal execution environment.

[0194] In some possible implementations, if the prompt entered by user 10 does not involve privacy data.

[0195] The second computing node 220 can also input the prompt word and intermediate calculation results to the decoding layer 221 of the large language model. The decoding layer 221 of the large language model uses the second ordinary operator 223 to calculate the prompt word and intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0196] As one possible implementation, the second computing node can also be connected to the storage node 300. The storage node 300 310 pre-deploys a vector database and a second security data packet 320.

[0197] The second computing node 210 can receive encrypted vector text sent by the storage node 300 and decrypt the encrypted vector text in the second trusted execution environment to obtain retrieval enhancement generation data. The encrypted vector text is encrypted by the second secure data packet 320, and the aforementioned retrieval enhancement generation data is determined by the knowledge database 312 based on prompts entered by the user.

[0198] The second computing node 210 can input the prompt word, intermediate calculation results, and retrieval enhancement generation data into the decoding layer 221 of the large language model. The decoding layer 221 of the large language model uses the second security operator 222 to calculate the prompt word, intermediate calculation results, and retrieval enhancement generation data to obtain the inference result corresponding to the prompt word.

[0199] It is understandable that if the prompt word entered by user 10 does not involve privacy data, the second computing node 210 can also input the prompt word, intermediate calculation results and retrieval enhancement generation data into the decoding layer 221 of the large language model. The pre-filling layer 221 of the large language model uses the second ordinary operator 223 to calculate the above prompt word, intermediate calculation results and retrieval enhancement generation data to obtain the inference result corresponding to the prompt word.

[0200] S730: Encrypt the inference result in the second trusted execution environment to obtain the second encrypted data and return the second encrypted data to the client.

[0201] The second computing node 210 can encrypt the inference results in the second trusted execution environment to obtain the second encrypted data, and then return the second encrypted data to the client 100.

[0202] It is understandable that after receiving the second encrypted data, the client 100 can use the first security toolkit 111 to decrypt the second encrypted data and obtain the reasoning result corresponding to the prompt words entered by the user.

[0203] In some possible implementations, if the prompt entered by user 10 does not involve privacy data.

[0204] The second computing node 210 can also use plaintext transmission to send the reasoning result corresponding to the prompt word to the client 100.

[0205] A second trusted execution environment is set up in the second computing node performing the decoding computation task in the PD separation architecture, and connected to the first computing node performing the decoding task through a secure data channel. The second computing node receives the prompt words and intermediate computation results sent by the first computing node through the secure data channel. In the second trusted execution environment, the second computing node uses a pre-deployed large language model decoding layer and a second secure operator to decode the user-input prompt words and intermediate computation results, obtaining the inference result corresponding to the prompt words. The inference result is then encrypted and sent to the client. This method ensures that the client's prompt words do not leave the secure area during the decoding process, achieves data security during transmission between the second computing node and the client, and protects the user's data privacy.

[0206] Thirdly, based on the content described above, a method for deploying a large language model provided in the embodiments of this application will be introduced. It is understood that this method is proposed based on the content described above, and some or all of the content of this method can be found in the description above.

[0207] For example, Figure 8 is a flowchart illustrating a method for deploying a large language model proposed in this application. It is understood that this method can be executed by the computing device 200 shown in Figure 3, or by other computing devices, such as any server, desktop computer, or laptop computer; this application does not limit the execution of such devices. Taking computing device 200 as an example, a third security data packet is pre-deployed in computing device 200. As shown in Figure 8, this method can achieve the deployment of the large language model through steps S810 to S840.

[0208] S810: Determine the pre-filling layer and decoding layer of the large language model based on the large language model.

[0209] First, the computing device 200 can split the large language model into a pre-filling layer and a decoding layer based on the task types of the pre-filling and decoding stages.

[0210] Then, the pre-filling layer 211 of the large language model can identify safe operators (first safe operators) that can be used for pre-filling computation through hardware-aligned sparse attention, dynamic gating mechanism and distributed alignment search.

[0211] Understandably, the decoding layer 221 of the large language model can also adopt the same approach to identify secure operators (second secure operators) that can be used for decoding computation.

[0212] S820: The computing device 200 can perform encryption using a pre-filled layer 211 of a large language model, a first security operator 212, a decoding layer 221 of the large language model, and a second security operator 222.

[0213] S830: The computing device 200 can send the encrypted pre-filled layer 211 and the first security operator 212 of the large language model to the first trusted execution environment of the first computing node 210, so that the first computing node 210 can decrypt the pre-filled layer 211 and the first security operator 222 of the large language model in the first trusted execution environment to complete the deployment of the pre-filled layer 211 and the first security operator 212 of the large language model.

[0214] S840: The computing device 200 can send the encrypted decoding layer 221 and the second security operator 222 of the large language model to the second trusted execution environment of the second computing node 210, so that the second computing node 210 can decrypt the decoding layer 221 and the second security operator 222 of the large language model in the second trusted execution environment to complete the deployment of the decoding layer 221 and the second security operator 222 of the large language model.

[0215] It is worth noting that in the above embodiments of this application, the computing device 200 may execute steps S830 and S840 simultaneously, or it may execute step S840 first and then step S830. In other words, this application does not limit the execution order of steps S830 and S840.

[0216] By pre-deploying the pre-filling layer and the first security operator of the large language model in the pre-filling phase of the execution stage in the first trusted execution environment of the first computing node, and pre-deploying the decoding layer and the second security operator of the large language model in the decoding phase of the execution stage in the second trusted execution environment of the second computing node, a reliable security foundation can be provided for subsequent data processing methods. This method ensures data security during the deployment of the large language model and security operators, improves data security during data inference, and protects user data privacy.

[0217] Fourthly, based on the content described above, a pre-filling processing apparatus provided in the embodiments of this application will be introduced. It is understood that this apparatus is proposed based on the content described above, and some or all of its contents can be found in the description above.

[0218] For example, Figure 9 is a schematic diagram of a pre-filling processing device proposed in this application. As shown in Figure 9, the pre-filling processing device 90 is applied to the first computing node 210 and connected to the second computing node 220 through a secure data interface 93. The device includes a first data security module 911 and a pre-filling module 912.

[0219] The first data security module 911 is used to receive the first encrypted data sent by the client 100, and decrypt the first encrypted data in the first trusted execution environment of the first computing node 210 to obtain the prompt words input by the user. The first trusted execution environment is used to deploy the pre-filled layer of the large language model and the first security operator.

[0220] The pre-filling module 912 is used to input the above-mentioned prompt word into the pre-filling layer of the large language model. The pre-filling layer of the large language model uses the first security operator to calculate the prompt word and obtain intermediate calculation results. The intermediate calculation results include the key-value cache corresponding to the above-mentioned prompt word and the next word.

[0221] The pre-filling module 912 is also used to send the prompt word and intermediate calculation results to the second computing node 220 through the secure data interface 93, so that the second computing node 220 can obtain the inference result corresponding to the prompt word based on the prompt word and the intermediate calculation results.

[0222] In some possible implementations, the pre-filling processing device 91 is also connected to the second computing node via a conventional data interface 94.

[0223] In this case, the first data security module 911 is also used to determine whether the prompt word involves the user's private data based on the prompt word.

[0224] This does not involve the user's privacy data.

[0225] The pre-filling module 912 is used to input the prompt words into the pre-filling layer of the large language model. The pre-filling layer of the large language model uses the first general operator to calculate the prompt words and obtain intermediate calculation results. The first general operator is an operator deployed in the first general execution environment of the first computing node 210.

[0226] The pre-filling module is also used to send the prompt words and intermediate calculation results to the second computing node 220 through the regular data channel 94, so that the second computing node 220 can perform calculations on the prompt words and intermediate calculation results to obtain the inference result corresponding to the prompt words.

[0227] Fifthly, based on the content described above, a decoding processing apparatus provided in the embodiments of this application will be introduced. It is understood that this apparatus is proposed based on the content described above, and some or all of its contents can be found in the description above.

[0228] Please refer to Figure 9. As shown in Figure 9, the decoding processing device 92 is applied to the second computing node 220 and is connected to the first computing node 210 through the secure data interface 93. The decoding processing device 92 includes a decoding module 921 and a second data security module 922.

[0229] The decoding module 921 is used to receive the prompt words and intermediate calculation results sent by the first computing node 210 through the secure data interface 93 in the second trusted execution environment of the second computing node 220. The intermediate calculation results include the key-value cache corresponding to the prompt words and the next word term. The second trusted execution environment is used to deploy the decoding layer and the second secure operator of the large language model.

[0230] The decoding module 921 is also used to input the prompt word and intermediate calculation results into the decoding layer of the large language model, and the decoding layer of the large language model uses the second security operator to calculate the prompt word and intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0231] The second data security module 922 is used to encrypt the inference result in the second trusted execution environment to obtain the second encrypted data, and return the second encrypted data to the client 100.

[0232] In some possible implementations, the decoding processing device 92 is also connected to the first computing node 210 via a conventional data interface 94.

[0233] In this case, the decoding module 921 is used to receive the prompt words and intermediate calculation results sent by the first computing node 210 through a regular data interface in the second general execution environment of the second computing node 220. The second general execution environment includes a second general operator.

[0234] The decoding module 921 is also used to input the prompt word and intermediate calculation results into the decoding layer of the large language model, and the decoding layer of the large language model uses the second ordinary operator to calculate the prompt word and intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0235] Sixthly, based on the content described above, a data processor system provided in the embodiments of this application will be introduced. It is understood that this system is proposed based on the content described above, and some or all of the content of this system can be found in the description above.

[0236] For example, Figure 10 is a schematic diagram of the system architecture of a data processor system proposed in this application. As shown in Figure 10, the data processing system 200 includes a first computing node 210 and a second computing node 220. The first computing node 210 and the second computing node 220 are interconnected through a secure data channel 23. The first computing node 210 includes a first trusted execution environment, in which a pre-filling layer 211 of a large language model and a first security operator 212 are pre-deployed. The second computing node 220 includes a second trusted execution environment, in which a decoding layer 221 of a large language model and a second security operator 222 are pre-deployed.

[0237] The first computing node 210 is used to receive the first encrypted data sent by the client 100, and decrypt the first encrypted data in the first trusted execution environment to obtain the prompt words input by the user 10.

[0238] The first computing node 210 is also used to input the prompt word into the pre-filling layer 211 of the large language model. The pre-filling layer 211 of the large language model uses the first security operator 212 to calculate the prompt word and obtain intermediate calculation results. The intermediate calculation results include the key-value cache corresponding to the prompt word and the next word. The prompt word and the intermediate calculation results are sent to the second computing node 220 through the secure data channel 23.

[0239] The second computing node 220 is used to input the prompt words and intermediate calculation results sent by the first computing node 210 into the decoding layer 221 of the large language model. The decoding layer 221 of the large language model uses the second security operator 222 to calculate the prompt words and intermediate calculation results to obtain the inference result corresponding to the prompt words.

[0240] The second computing node 220 is also used to encrypt the above inference results in the second trusted execution environment to obtain second encrypted data, and return the second encrypted data to the client 100.

[0241] In some possible implementations, the first computing node 210 further includes a first general execution environment. The second computing node includes a second general execution environment, and the first general execution environment is connected to the second general execution environment via a conventional data channel 24. The first general execution environment is used to deploy the first general operator 213, and the second general execution environment is used to deploy the second general operator 223.

[0242] In this case, the data processing system 200 can process the prompt words according to whether they involve the user's privacy data.

[0243] The first computing node 210 is also used to determine whether the prompt word involves the user's privacy data based on the above prompt word.

[0244] If the prompt does not involve the user's privacy data, the prompt is input into the pre-filling layer 211 of the large language model. The pre-filling layer 211 of the large language model uses the first ordinary operator 213 to calculate the prompt and obtain the intermediate calculation result.

[0245] The first computing node 210 can send the prompt word and intermediate calculation results to the second computing node 220 through the regular data channel 24.

[0246] The second computing node 220 is also used to input the prompt word and intermediate calculation results to the decoding layer 221 of the large language model. The decoding layer 221 of the large language model uses the second ordinary operator 223 to calculate the prompt word and intermediate calculation results to obtain the inference result corresponding to the prompt word.

[0247] In addition to the methods, apparatus, and electronic devices described above, embodiments of this application may also provide a computer program product, comprising computer program instructions. When executed by a processor, the computer program instructions cause the processor to perform the steps of the methods described in the "Methods" section of this specification. The computer program product can be written in any combination of one or more programming languages ​​to execute the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The computer program code can be in source code form, object code form, executable file, or some intermediate form. The computer program code can be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0248] This application embodiment may also provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to the various embodiments of this disclosure described in the "Method" section above. The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. It should be noted that the content of the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, a computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0249] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0250] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0251] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0252] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0253] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0254] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

[0255] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The specific embodiments described above have further detailed the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0256] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method is applied to a first computing node, which includes a first trusted execution environment connected to a second computing node via a secure data channel. The first trusted execution environment is used to deploy a pre-filled layer of a large language model and a first secure operator. The method includes: Receive the first encrypted data sent by the client, and decrypt the first encrypted data in the first trusted execution environment to obtain the prompt words entered by the user; The prompt word is input into the pre-filling layer of the large language model, and the pre-filling layer of the large language model uses the first security operator to calculate the prompt word to obtain an intermediate calculation result. The intermediate calculation result includes the key-value cache and the next word element corresponding to the prompt word. The prompt word and the intermediate calculation result are sent to the second computing node through the secure data channel, so that the second computing node can obtain the inference result corresponding to the prompt word based on the prompt word and the intermediate calculation result.

2. The method according to claim 1, characterized in that, The client includes a pre-deployed first security toolkit, which is used to encrypt the prompt word.

3. The method according to claim 1, characterized in that, The first computing node includes a first general execution environment, which is used to deploy a first general operator. The first computing node is connected to a second computing node through a conventional data channel. After the first trusted execution environment decrypts the first encrypted data to obtain the prompt word, the process further includes: Based on the prompt word, determine whether the prompt word involves the user's privacy data; If the prompt does not involve the user's privacy data, the prompt is input into the pre-filling layer of the large language model, and the pre-filling layer of the large language model uses the first ordinary operator to calculate the prompt to obtain the intermediate calculation result. The prompt word and the intermediate calculation result are sent to the second calculation node through the conventional data channel, so that the second calculation node can perform calculations on the prompt word and the intermediate calculation result to obtain the reasoning result corresponding to the prompt word.

4. The method according to claim 1, characterized in that, The first computing node is connected to a storage node, which pre-deploys a knowledge database and a second security data packet. The prompt word is input into the pre-filling layer of the large language model, and the pre-filling layer uses the first security operator to calculate the prompt word, obtaining intermediate calculation results, including: The system receives encrypted vector text sent by the storage node and decrypts the encrypted vector text in the first trusted execution environment to obtain search enhancement generation data. The encrypted vector text is encrypted by the second secure data packet, and the search enhancement generation data is determined by the knowledge database based on the prompt words. The prompt words and the search enhancement generated data are input into the pre-filling layer of the large language model. The pre-filling layer of the large language model uses the first security operator to calculate the prompt words and the search enhancement generated data to obtain intermediate calculation results.

5. A data processing method, characterized in that, The method is applied to a second computing node, which includes a second trusted execution environment connected to a first computing node via a secure data channel. This second trusted execution environment is used to deploy a decoding layer for a large language model and a second secure operator. Receive prompt words and intermediate calculation results sent by the first computing node through the secure data channel, wherein the intermediate calculation results include the key-value cache corresponding to the prompt words and the next word element; The prompt word and the intermediate calculation result are input into the decoding layer of the large language model. The decoding layer of the large language model uses the second security operator to calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word. The inference result is encrypted in the second trusted execution environment to obtain second encrypted data, and the second encrypted data is returned to the client.

6. The method according to claim 5, characterized in that, The client includes a pre-deployed security toolkit, which is used to decrypt the second encrypted data to obtain the reasoning result corresponding to the prompt word.

7. The method according to claim 5, characterized in that, The second computing node includes a second general execution environment, which is connected to the first computing node via a conventional data channel. The second general execution environment is used to deploy second general operators. After receiving the prompt words and intermediate calculation results sent by the first computing node, the process further includes: Receive the prompt word and the intermediate calculation result sent by the first computing node through the conventional data channel; The prompt word and the intermediate calculation result are input into the decoding layer of the large language model. The decoding layer of the large language model uses the second ordinary operator to calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word.

8. The method according to claim 5, characterized in that, The first computing node is connected to a storage node. The storage node is pre-deployed with a knowledge database and a second security data packet. The prompt word and the intermediate calculation result are input to the decoding layer of the large language model. The decoding layer of the large language model uses the second security operator to calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word, including: The system receives encrypted vector text sent by the storage node and decrypts the encrypted vector text in the second trusted execution environment to obtain search enhancement generation data. The encrypted vector text is encrypted by the second secure data packet, and the search enhancement generation data is determined by the knowledge database based on the prompt words. The prompt word, the intermediate calculation result, and the search enhancement generated data are input into the decoding layer of the large language model. The decoding layer of the large language model uses the second security operator to calculate the prompt word, the intermediate calculation result, and the search enhancement generated data to obtain the inference result corresponding to the prompt word.

9. A method for deploying a large language model, characterized in that, The method is applied to a computing device, the computing device including a third security toolkit, and the method includes: The pre-filling layer and the decoding layer of the large language model are determined based on the large language model. The third security toolkit is used to encrypt the pre-filled layer, the first security operator, the decoding layer of the large language model, and the second security operator. The encrypted pre-filled layer of the large language model and the first security operator are sent to the first trusted execution environment of the first computing node, so that the first computing node can decrypt the pre-filled layer of the large language model and the first security operator in the first trusted execution environment. The encrypted decoding layer of the large language model and the second security operator are sent to the second trusted execution environment of the second computing node, so that the second computing node can decrypt the decoding layer of the large language model and the second security operator in the second trusted execution environment.

10. A pre-filling treatment apparatus, characterized in that, The device is applied to a first computing node and connected to a second computing node via a secure data interface. The device includes: The first data security module is used to receive first encrypted data sent by the client and decrypt the first encrypted data in the first trusted execution environment of the first computing node to obtain the prompt words input by the user. The first trusted execution environment is used to deploy the pre-filling layer of the large language model and the first security operator. A pre-filling module is used to input the prompt word into the pre-filling layer of the large language model. The pre-filling layer of the large language model uses the first security operator to calculate the prompt word and obtain an intermediate calculation result. The intermediate calculation result includes the key-value cache corresponding to the prompt word and the next word. The prompt word and the intermediate calculation result are sent to the second computing node through the secure data interface so that the second computing node can obtain the inference result corresponding to the prompt word based on the prompt word and the intermediate calculation result.

11. The apparatus according to claim 10, characterized in that, The device is also connected to a second computing node via a conventional data interface, and the first data security module is further used for: Based on the prompt word, determine whether the prompt word involves the user's privacy data; The pre-filling module is also used for: If the prompt does not involve the user's privacy data, the prompt is input into the pre-filling layer of the large language model, and the pre-filling layer of the large language model calculates the prompt using a first general operator to obtain the intermediate calculation result; the first general operator is an operator deployed in the first general execution environment of the first computing node; The prompt word and the intermediate calculation result are sent to the second calculation node through the conventional data channel, so that the second calculation node can perform calculations on the prompt word and the intermediate calculation result to obtain the reasoning result corresponding to the prompt word.

12. A decoding processing device, characterized in that, The device is applied to a second computing node and is connected to a first computing node via a secure data interface. The device includes: A decoding module is configured to receive, within the second trusted execution environment of the second computing node, a prompt word and intermediate calculation results sent by the first computing node through the secure data interface. The intermediate calculation results include a key-value cache and the next word element corresponding to the prompt word. The second trusted execution environment is configured to deploy a decoding layer of a large language model and a second secure operator. The prompt word and the intermediate calculation results are input into the decoding layer of the large language model, whereby the decoding layer uses the second secure operator to calculate the prompt word and the intermediate calculation results to obtain the inference result corresponding to the prompt word. The second data security module is used to encrypt the inference result in the second trusted execution environment to obtain second encrypted data, and then return the second encrypted data to the client.

13. The apparatus according to claim 12, characterized in that, The device is also connected to the first computing node via a conventional data interface, and the decoding module is further used for: The second computing node receives the prompt word and intermediate calculation result sent by the first computing node through the regular data interface in the second normal execution environment. The second normal execution environment includes a second normal operator. The prompt word and the intermediate calculation result are input to the decoding layer of the large language model. The decoding layer of the large language model uses the second normal operator to calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word.

14. A data processing system, characterized in that, The system includes a first computing node and a second computing node. The first computing node includes a first trusted execution environment, which pre-deploys a pre-filling layer of a large language model and a first security operator. The second computing node includes a second trusted execution environment, which pre-deploys a decoding layer of a large language model and a second security operator. The first trusted execution environment and the second trusted execution environment are interconnected through a secure data channel. The first computing node is used for: Receive the first encrypted data sent by the client, and decrypt the first encrypted data in the first trusted execution environment to obtain the prompt words entered by the user; The prompt word is input into the pre-filling layer of the large language model, and the pre-filling layer of the large language model uses the first security operator to calculate the prompt word to obtain an intermediate calculation result. The intermediate calculation result includes the key-value cache and the next word element corresponding to the prompt word. The prompt word and the intermediate calculation result are sent to the second calculation node through the secure data channel; The second computing node is used for: The prompt word and the intermediate calculation result are input into the decoding layer of the large language model. The decoding layer of the large language model uses the second security operator to calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word. The inference result is encrypted in the second trusted execution environment to obtain the second encrypted data, and the second encrypted data is returned to the client.

15. The system according to claim 14, characterized in that, The first computing node includes a first general execution environment for deploying a first general operator. The second computing node includes a second general execution environment for deploying a second general operator. The first computing node is connected to the second computing node via a conventional data channel. The first computing node is also used for: Based on the prompt word, determine whether the prompt word involves the user's privacy data; If the prompt does not involve the user's privacy data, the prompt is input into the pre-filling layer of the large language model, and the pre-filling layer of the large language model uses the first ordinary operator to calculate the prompt to obtain the intermediate calculation result. The prompt word and the intermediate calculation result are sent to the second calculation node through the conventional data channel; The second computing node is also used for: The prompt word and the intermediate calculation result are input into the decoding layer of the large language model. The decoding layer of the large language model uses the second ordinary operator to calculate the prompt word and the intermediate calculation result to obtain the inference result corresponding to the prompt word.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a first computing node, cause the first computing node to perform the data processing method according to any one of claims 1-4.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the first computing node, the first computing node performs the data processing method as described in any one of claims 1-4.

18. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a second computing node, cause the second computing node to perform the data processing method according to any one of claims 5-8.

19. A computer program product containing instructions, characterized in that, When the instruction is executed by the second computing node, the second computing node performs the data processing method as described in any one of claims 5-8.

Images