Data processing method and related device

By splitting the prompt words and vectorizing them, and using database matching of similar vectors to reuse the key-value cache, the problem of low key-value cache reuse rate in the Transformer model is solved, thus improving inference efficiency and overall efficiency.

WO2026237688A1PCT designated stage Publication Date: 2026-11-19HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/092552
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2026-04-23
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

In existing Transformer models, the reuse of key-value (KV) caches can only achieve exact matching. It cannot reuse historical KV caches when the input prompt word is slightly different from the historical prompt word, resulting in increased redundant calculations and low inference efficiency.

Method used

By splitting the prompt words into multiple sub-prompt words, and vectorizing each sub-prompt word, similarity matching is performed using a database to find similar vectors for reuse of the key-value cache.

Benefits of technology

It reduces unnecessary redundant calculations, improves inference efficiency, and enables fuzzy matching, thereby increasing the reuse rate of historical key-value caches.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026092552_19112026_PF_FP_ABST
    Figure CN2026092552_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A data processing method. The method comprises: a data processing device first acquiring a prompt of a model inference task; upon acquiring the prompt, the data processing device splitting the prompt to obtain a plurality of sub-prompts; then, performing vectorization processing on each of the plurality of sub-prompts to obtain a plurality of first vectors having one-to-one correspondence to the plurality of sub-prompts; performing similarity matching in a database on the basis of the plurality of first vectors, to obtain at least one second vector that matches at least one first vector among the plurality of first vectors in one-to-one correspondence; and finally, the data processing device determining a processing result of the model inference task on the basis of a KV cache corresponding to the at least one second vector.
Need to check novelty before this filing date? Find Prior Art

Description

A data processing method and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202510633045.0, filed on May 15, 2025, entitled "A Data Processing Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a data processing method and related equipment. Background Technology

[0003] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0004] To reduce redundant computations in Transformer models, key-value (KV) caching is commonly used. Specifically, in a Transformer model, the query, key, and value vectors need to be calculated for each inference iteration. KV caching involves caching the previously calculated K and V matrices, allowing them to be directly used in subsequent inference processes without recalculation. For example, suppose the inference task is to generate the sentence "It is a cat." When generating the first word "It," the model calculates and caches the Q, K, and V matrices for "It." When generating the next word "is," the model only needs to calculate the Q matrix for "is" and uses the previously cached K and V matrices for "It" to calculate the self-attention score. This continues, with each new token generated using the cached K and V matrices, thus reducing unnecessary redundant computations.

[0005] Key-value (KV) caches are often reused using hash code matching. Specifically, when the input prompt word matches a previous prompt word, the previous KV cache can be reused through hash code matching. However, this method can only achieve exact matching; it cannot reuse the previous KV cache when the input prompt word differs slightly from the previous prompt word. Summary of the Invention

[0006] This application provides a data processing method and related equipment. The method vectorizes multiple sub-prompts corresponding to a prompt word and searches for similar vectors, allowing the use of key-value (KV) caches corresponding to these similar vectors during inference. This not only reduces unnecessary redundant calculations and improves inference efficiency but also enables fuzzy matching, increasing the reuse rate of historical KV caches.

[0007] This application provides a data processing method in its first aspect. This method is executed by a data processing device, or by a component (e.g., a processor, chip, or chip system) within the data processing device, or by a logic module or software capable of implementing all or part of the functions of the data processing device. In this first aspect and its possible implementations, the method is described using an example of execution by a data processing device. In this method, the data processing device first acquires prompt words for a model inference task. After acquiring the prompt words, the data processing device splits them into multiple sub-prompt words. Each sub-prompt word is then vectorized to obtain multiple first vectors corresponding one-to-one with the sub-prompt words. Similarity matching is then performed on the multiple first vectors in a database to obtain at least one second vector whose meaning matches at least one of the multiple first vectors. Finally, the data processing device determines the processing result of the model inference task based on the key-value caches corresponding to each of the at least one second vector.

[0008] Among them, the similarity between a matching first vector and a second vector is greater than or equal to a threshold. The database includes multiple second vectors, and each second vector has its own KV cache.

[0009] Optionally, the second vector corresponding to a KV cache can be understood as the second vector corresponding to a prompt word identifier, which is associated with a KV cache. That is, the data processing device can determine the prompt word identifier corresponding to the second vector and retrieve the corresponding KV cache by looking up the prompt word identifier in a table. The table stores the association between the prompt word identifier and the KV cache.

[0010] In this scheme, multiple sub-hints corresponding to the hint words are vectorized, and similarity matching is performed to find similar vectors. This allows the use of key-value (KV) caches corresponding to similar vectors during inference. This not only reduces unnecessary redundant calculations and improves inference efficiency but also enables fuzzy matching, increasing the reuse rate of historical KV caches.

[0011] In one possible implementation, the data processing device can also acquire historical prompt words for the historical model reasoning task, the historical prompt words including multiple historical sub-prompt words; calculate the KV cache corresponding to each of the multiple historical sub-prompt words; perform vectorization processing on at least one historical sub-prompt word of the multiple historical sub-prompt words to obtain at least one second vector corresponding one-to-one with at least one historical sub-prompt word; and store at least one second vector and the KV cache corresponding to each of the at least one second vector in the database.

[0012] In this possible implementation, after the data processing device calculates the KV cache corresponding to each historical prompt word, it performs vectorization processing to obtain at least one second vector corresponding to at least one historical sub-prompt word, and then stores the at least one second vector and its corresponding KV cache in the database. This allows the data processing device to perform similarity matching in the database based on the first vector related to the model inference task when calculating the model inference task, thus reusing the historical KV cache, reducing unnecessary redundant calculations, and improving the inference efficiency of the model inference task.

[0013] In one possible implementation, the data processing device specifically splits the prompt word into multiple sub-prompt words based on the block size of the KV cache.

[0014] In this possible implementation, the prompt words are split according to the block size of the KV cache, which can adapt to the KV cache rules and improve the accuracy of subsequent reuse of historical KV cache.

[0015] In one possible implementation, the number of token identifiers corresponding to each sub-prompt word in the multiple sub-prompt words is the same as the block size.

[0016] In this possible implementation, the number of tokens corresponding to each sub-prompt word is the same as the block size, reducing management complexity.

[0017] In one possible implementation, the difference in the number of tokens corresponding to any two adjacent sub-prompt words is the block size.

[0018] In this possible implementation, the relevance between sub-prompts is improved by leveraging the inclusion relationship between the subsequent sub-prompt and the preceding sub-prompt.

[0019] In one possible implementation, one of the multiple sub-prompt words has a position index, which indicates the position of a sub-prompt word among the multiple sub-prompt words.

[0020] In this possible implementation, the relevance between sub-prompts can be improved by using the position index of the sub-prompts.

[0021] In one possible implementation, each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. Specifically, the data processing device performs similarity matching in the database based on the position index sorting of multiple first vectors to obtain at least one second vector. At least one second vector has the same prompt word identifier.

[0022] In this possible implementation, sequential retrieval based on the position index can ensure that multiple retrieved second vectors correspond to the same historical prompt word as much as possible, thereby improving the accuracy of subsequent reuse of the KV cache.

[0023] In one possible implementation, each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. Specifically, the data processing device performs similarity matching from the database based on the position index of multiple first vectors to obtain at least one second vector. At least one second vector has the same prompt word identifier.

[0024] In this possible implementation, the retrieval order does not need to be considered, thus improving retrieval efficiency.

[0025] In one possible implementation, one of the multiple first vectors has a prompt word identifier, which is used to identify the prompt word to which the first vector belongs; the data processing device can also store multiple first vectors in the database based on the prompt word identifier of the first vector, and the multiple first vectors correspond to a KV cache.

[0026] In this possible implementation, after the model inference task is completed, the vectors and key-value caches related to the model inference task can be stored to update the database for future inference.

[0027] In one possible implementation, the number of multiple first vectors is greater than or equal to the number of at least one second vector.

[0028] In this possible implementation, if each of the multiple first vectors can retrieve a second vector and the retrieved second vectors correspond to the same historical suggestion word, then the number of multiple first vectors is the same as the number of multiple second vectors. If there is a first vector among the multiple first vectors that fails to retrieve a second vector, then the number of multiple first vectors is greater than the number of multiple second vectors, meaning that at least one first vector has no search results.

[0029] A second aspect of this application provides a data processing device, which can be a cloud device or a terminal device, or the device can be a component (e.g., a processor, chip, or chip system) in the cloud device or terminal device, or the device can be a logic module or software capable of implementing all or part of the functions of the data processing device. The communication device includes an acquisition module and a processing module.

[0030] The acquisition module is used to acquire prompt words for model inference tasks;

[0031] The processing module is used to split the prompt words into multiple sub-prompt words;

[0032] The processing module is also used to vectorize each of the multiple sub-prompt words to obtain multiple first vectors that correspond one-to-one with the multiple sub-prompt words;

[0033] The processing module is also used to perform similarity matching in the database based on multiple first vectors to obtain at least one second vector that matches at least one of the multiple first vectors. The similarity between a matching first vector and a second vector is greater than or equal to a threshold. The database includes multiple second vectors, and each second vector has its own KV cache.

[0034] The processing module is also used to determine the processing result of the model inference task based on the KV cache corresponding to at least one second vector.

[0035] In one possible implementation, the acquisition module is further configured to acquire historical prompt words for the historical model inference task, the historical prompt words including multiple historical sub-prompt words; the processing module is further configured to calculate the KV cache corresponding to each of the multiple historical sub-prompt words; the processing module is further configured to perform vectorization processing on at least one of the multiple historical sub-prompt words to obtain at least one second vector corresponding one-to-one with at least one historical sub-prompt word; the processing module is further configured to store at least one second vector and the KV cache corresponding to each of the at least one second vector in the database.

[0036] In one possible implementation, the processing module is specifically used to split the prompt words into multiple sub-prompt words based on the block size of the KV cache.

[0037] In one possible implementation, the number of token identifiers corresponding to each sub-prompt word in the multiple sub-prompt words is the same as the block size.

[0038] In one possible implementation, the difference in the number of tokens corresponding to any two adjacent sub-prompt words is the block size.

[0039] In one possible implementation, one of the multiple sub-prompt words has a position index, which indicates the position of a sub-prompt word among the multiple sub-prompt words.

[0040] In one possible implementation, each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. The processing module is specifically used to perform similarity matching in the database in a sequential order based on the position index of multiple first vectors to obtain at least one second vector, and at least one second vector has the same prompt word identifier.

[0041] In one possible implementation, each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. The processing module is specifically used to perform similarity matching from the database based on the position index of multiple first vectors to obtain at least one second vector, and at least one second vector has the same prompt word identifier.

[0042] In one possible implementation, one of the multiple first vectors has a prompt word identifier, which is used to identify the prompt word to which the first vector belongs; the processing module is also used to store multiple first vectors in the database based on the prompt word identifier of the first vector, and the multiple first vectors correspond to a KV cache.

[0043] In one possible implementation, the number of multiple first vectors is greater than or equal to the number of at least one second vector.

[0044] A third aspect of this application provides a data processing apparatus, including at least one processor coupled to a memory; the memory is used to store a program or instructions; the at least one processor is used to execute the program or instructions to enable the data processing apparatus to implement any possible implementation of the first aspect.

[0045] The fourth aspect of this application provides a data processing device, including at least one logic circuit and an input / output interface; the logic circuit is used to perform a method as described in any of the possible implementations of the first aspect above.

[0046] The fifth aspect of this application provides a computer-readable storage medium for storing one or more computer-executable instructions, which, when executed by a processor, perform a method as described in any possible implementation of any of the first aspects above.

[0047] The sixth aspect of this application provides a computer program product in which, when a computer program in the computer program product is executed by a processor, the processor executes a method of any possible implementation of any aspect of the first aspect described above.

[0048] The seventh aspect of this application provides a chip or chip system including at least one processor for supporting a method of implementing any possible implementation of any of the first aspects described above in a communication device.

[0049] In one possible design, the chip system may further include a memory for storing program instructions and data necessary for the communication device. The chip system may be composed of chips or may include chips and other discrete devices. Optionally, the chip system may also include interface circuitry that provides program instructions and / or data to at least one processor.

[0050] An eighth aspect of this application provides a data processing apparatus including a chip system as described in the seventh aspect above. The chip system includes a processor and a communication interface for communicating with a module outside the chip system. The processor is used to run computer programs or instructions, enabling the data processing apparatus to perform the methods of any of the above aspects.

[0051] A ninth aspect of this application provides a computing device cluster, including at least one data processing device as described in the second or eighth aspect, wherein any one data processing device is used to run a computer program or instructions, enabling the computing device cluster to perform the methods of any of the above aspects. Alternatively, some or all of the data processing devices may be used together to run a computer program or instructions, enabling the computing device cluster to perform the methods of any of the above aspects.

[0052] The technical effects of any of the design methods in aspects two through nine can be found in the technical effects of the different design methods in aspect one above, and will not be repeated here.

[0053] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0054] Figure 1 is a schematic diagram of the system architecture provided in this application;

[0055] Figure 2 is a schematic diagram of a chip hardware structure provided in this application;

[0056] Figure 3A is a structural diagram of the data processing system provided in this application;

[0057] Figure 3B is another structural schematic diagram of the data processing system provided in this application;

[0058] Figure 4 is a schematic diagram of the three stages involved in the data processing provided in this application;

[0059] Figure 5 is a flowchart illustrating the data processing method provided in this application;

[0060] Figure 6 is an example of a split prompt word provided in this application;

[0061] Figure 7 is another example of the split prompt words provided in this application;

[0062] Figure 8 is an example diagram of obtaining the first vector provided in this application;

[0063] Figure 9 is another example diagram of obtaining the first vector provided in this application;

[0064] Figure 10 is an example diagram of serial retrieval provided in this application;

[0065] Figure 11 is an example diagram of parallel retrieval provided in this application;

[0066] Figures 12 to 14 are several schematic diagrams of the data processing equipment involved in this application;

[0067] Figure 15 is a schematic diagram of the structure of a data processing equipment cluster provided in this application;

[0068] Figure 16 is a schematic diagram of another data processing equipment cluster provided in this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0070] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of this application will be introduced below.

[0071] 1. Large Language Model (LLM)

[0072] LLM (Language Learning Model) is a model based on machine learning and natural language processing techniques. It learns to perform human-like language understanding and generation by training on large amounts of text data. The core idea of ​​LLM is to learn patterns and structures of natural language through large-scale unsupervised training, which to some extent simulates the human language cognition and generation process. Compared to traditional Natural Language Processing (NLP) models, LLM can better understand and generate natural text, while also demonstrating a certain level of logical thinking and reasoning ability.

[0073] Optionally, the LLM inference phase is divided into two processes: prefill and decode. Prefill is the process from when the user inputs the prompt to when the first token is generated, while decode is the process from when the first token is generated to when inference stops.

[0074] 2. Key-Value Cache

[0075] Key-value (KV) caching can be applied to Transformer models. Specifically, in Transformer models, query, key, and value vectors need to be calculated for each inference step. KV caching refers to caching the previously calculated K and V matrices. In subsequent inference processes, the cached KV matrices can be used directly without recalculation. For example, suppose the inference task is to generate the sentence "It is a cat". When generating the first word "It", the model calculates and caches the Q, K, and V matrices for "It". When generating the next word "is", the model only needs to calculate the Q matrix for "is" and uses the previously cached K and V matrices for "It" to calculate the self-attention score. This continues, and when generating each new token, the cached K and V matrices can be used, thus reducing unnecessary recalculation.

[0076] 3. Key-Value (KV) cache blocks

[0077] The model allows storing contiguous key-value vectors in non-contiguous memory space. Specifically, the model can divide the KV cache of each sequence into KV blocks, with each KV block containing a fixed number of tokens (or KV block size).

[0078] Optionally, after the model obtains the prompt words, it divides the prompt words into logical blocks according to the set block size. After dividing the logical blocks, they are mapped to physical blocks. The physical blocks are where the actual key-value values ​​are stored, and the relationship (or mapping relationship) between logical blocks and physical blocks can be recorded through a block table. Each row of the block table records the physical block corresponding to the logical block and the number of filling positions.

[0079] 4. Word embedding

[0080] Embedding is a technique that maps data or information from a high-dimensional space to a low-dimensional space. In NLP, embedding typically refers to mapping text or words to representations in a real-valued vector space. It captures the semantic information of text or words into vectors, enabling computers to better understand and process textual data. Embeddings can be generated using various methods, the most common being the use of neural network models, such as word embeddings or text embeddings. These models learn a mapping from text to a vector space, where similar text will have similar vector representations in the embedding space. Embedding plays a crucial role in many NLP tasks, such as text classification, named entity recognition, sentiment analysis, and machine translation. By mapping text to continuous vector representations, embedding techniques enable computers to better understand and reason about textual data.

[0081] Currently, key-value (KV) cache reuse often employs hash code matching. Specifically, when the input prompt word matches a historical prompt word, the historical KV cache can be reused through hash code matching. However, this method can only achieve exact matching; it cannot reuse the historical KV cache when the input prompt word differs slightly from the historical prompt word.

[0082] To address the aforementioned technical problems, this application provides a data processing method and related equipment. The method vectorizes multiple sub-hints corresponding to a hint word and searches for similar vectors, allowing the use of the corresponding key-value cache during inference. This not only reduces unnecessary redundant calculations and improves inference efficiency but also enables fuzzy matching, increasing the reuse rate of historical key-value caches.

[0083] Before describing the data processing method and related equipment of the embodiments of this application in conjunction with the accompanying drawings, the system architecture provided in the embodiments of this application will be explained first.

[0084] Referring to Figure 1, this application embodiment provides a system architecture 100. As shown in system architecture 100, the data acquisition device 160 is used to collect training data. In this application embodiment, the training data includes: a model inference task and the processing result of the model inference task. The model inference task is used to describe user needs, and the processing result is the response information made in response to user needs. The presentation forms of the model inference task and the processing result can be text, images, audio, video, etc., which are not specifically limited here. The training data is stored in a database 130, and the training device 120 trains a target model / rule 101 based on the training data maintained in the database 130. The following will describe in more detail how the training device 120 obtains the target model / rule 101 based on the training data. This target model / rule 101 can be used to implement the data processing method provided in this application embodiment. The target model / rule 101 in this application embodiment may specifically include a neural network (or model, AI model, etc.). It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. It should also be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training. The above description should not be construed as a limitation on the embodiments of this application.

[0085] The target model / rule 101 trained by training device 120 can be applied to different systems or devices, such as execution device 110 shown in Figure 1. Execution device 110 can be a terminal, such as a mobile terminal, tablet computer, laptop computer, virtual reality (VR), augmented reality (AR), vehicle terminal, etc., or it can be a server or cloud. In Figure 1, execution device 110 is configured with I / O interface 112 for data interaction with external devices. Users can input data to I / O interface 112 through client device 140. The input data in this embodiment can include model inference tasks, etc. Furthermore, the input data can be user input, uploaded by the user through a camera device, received from other devices, or even come from a database; specific details are not limited here.

[0086] The preprocessing module 113 is used to preprocess the input data received by the I / O interface 112 (e.g., including at least one of the following: noise reduction, padding, etc.), which is not specifically limited here.

[0087] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0088] Finally, I / O interface 112 returns the processing result, as described above, to client device 140, thereby providing it to the user.

[0089] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0090] In the scenario shown in Figure 1, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0091] It is worth noting that Figure 1 is only a schematic diagram of a system architecture provided by an embodiment of this application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0092] The following describes a chip hardware structure provided by an embodiment of this application.

[0093] Figure 2 illustrates a chip hardware structure provided in an embodiment of this application. The chip includes a neural network processor 20. This chip can be installed in the execution device 110 shown in Figure 1 to perform the computational work of the computing module 111. The chip can also be installed in the training device 120 shown in Figure 1 to perform the training work of the training device 120 and output the target model / rule 101.

[0094] The neural network processor 20 can be any processor suitable for large-scale XOR operations, such as a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). Taking an NPU as an example: the neural network processor 20 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 203, and the controller 204 controls the arithmetic circuit 203 to retrieve data from the memory (weight memory or input memory) and perform operations.

[0095] In some implementations, the arithmetic circuit 203 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 203 is a two-dimensional pulsating array. The arithmetic circuit 203 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 203 is a general-purpose matrix processor.

[0096] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 202 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 201 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 208.

[0097] The vector computation unit 207 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit 207 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0098] In some implementations, vector computation unit 207 stores the processed output vector in unified memory 206. For example, vector computation unit 207 can apply a nonlinear function to the output of arithmetic circuit 203, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 207 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 203, for example, for use in subsequent layers of a neural network.

[0099] The unified memory 206 is used to store input data and output data.

[0100] The weight data is directly transferred from the external memory to the input memory 201 and / or the unified memory 206 through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory 202, and the data in the unified memory 206 is stored in the external memory.

[0101] The bus interface unit (BIU) 210 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 209 via the bus.

[0102] The instruction fetch buffer 209, which is connected to the controller 204, is used to store the instructions used by the controller 204.

[0103] The controller 204 is used to call the instructions cached in the instruction fetch memory 209 to control the operation of the computing accelerator.

[0104] Generally, the unified memory 206, input memory 201, weight memory 202, and instruction fetch memory 209 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0105] The following sections introduce several data processing system scenarios involved in this application.

[0106] Figure 3A is a schematic diagram of a data processing system provided in an embodiment of this application. The data processing system includes a terminal device (Figure 3A only uses a mobile phone as an example) and a data processing device. It is understood that the terminal device can be not only a mobile phone, but also a tablet computer, portable game console, PDA, laptop computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, in-vehicle media playback device, wearable electronic device, VR, AR, vehicle, in-vehicle terminal, aircraft terminal, intelligent robot, and other terminal devices. The terminal device is the initiator of data processing; as the initiator of data processing requests, requests are typically initiated by the user through the terminal device.

[0107] The aforementioned data processing equipment can be cloud servers, network servers, application servers, management servers, or other devices or servers with data processing capabilities. The data processing equipment receives data processing requests from terminal devices through an interactive interface, and then performs data processing methods such as machine learning, deep learning, search, reasoning, and decision-making through a storage device and a data processing processor. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0108] In the data processing system shown in Figure 3A, the terminal device can receive user instructions. For example, the terminal device can obtain the user-input / selected model inference task, and then send a request to the data processing device, causing the data processing device to perform data processing on the model inference task obtained by the terminal device, thereby obtaining the processing result corresponding to the model inference task. For instance, the terminal device can obtain the user-input question, and then send a request to the data processing device, causing the data processing device to infer the question, thereby obtaining the processing result, and displaying the processing result through the terminal device for the user to view and use.

[0109] In Figure 3A, the data processing device can be understood as a cloud device used to execute the data processing method of the embodiments of this application.

[0110] Figure 3B is another structural schematic diagram of the data processing system provided in the embodiment of this application. In Figure 3B, the terminal device (in Figure 3B, only the terminal device is a mobile phone as an example) directly serves as the data processing device. The terminal device can directly obtain the model inference task and process it directly by the hardware of the terminal device itself. The specific process is similar to that in Figure 3A, and can be referred to the above description, which will not be repeated here.

[0111] Optionally, in the data processing system shown in Figure 3B, the terminal device can receive user instructions. For example, the terminal device can obtain the model inference task input by the user in the terminal device, and then the terminal device itself performs data processing on the model inference task to obtain the processing result corresponding to the model inference task, and display the processing result for the user to view and use.

[0112] In Figure 3B, the data processing device can be understood as a terminal device, which can execute the data processing method of the embodiments of this application itself.

[0113] The terminal devices in Figures 3A and 3B can be the client device 140 or the execution device 110 in Figure 1. The data processing device in Figure 3A can be the execution device 110 in Figure 1. The data storage system 150 can store the data to be processed by the execution device 110. The data storage system 150 can be integrated on the execution device 110 or set up on the cloud or other network servers.

[0114] The processors in Figures 3A and 3B can perform data training / machine learning / deep learning through neural network models, and use the trained or learned models to perform data processing on the problem, thereby obtaining the corresponding processing results.

[0115] It is understood that data processing devices and / or terminal devices can be implemented not only as physical devices as described above, but also as at least one computing instance within a virtual machine or container. When a data processing device is implemented using a virtual machine or container, it actually exists as a cloud computing product, capable of providing cloud services. Furthermore, a data processing device can be implemented using multiple computing instances of the same type. For example, a data processing device can be implemented using multiple physical hosts, multiple virtual machines, or multiple containers.

[0116] It should be noted that multiple compute instances can be distributed within the same region or across different regions. Furthermore, multiple compute instances can be distributed within the same availability zone (AZ) or across different AZs, with each AZ comprising one or more geographically proximate data centers. Typically, a single region can include multiple AZs.

[0117] Similarly, multiple compute instances can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0118] Please refer to Figure 4, which shows another system architecture provided in this application embodiment. It is understood that the multiple stages shown in Figure 4 can be executed by the preprocessing module 113 in Figure 1, or by the target model / rule 101 in Figure 1, or the preprocessing module 113 can execute some stages, and then the target model / rule 101 can execute other stages. The specifics are not limited here.

[0119] For example, the model acquires inference tasks and obtains prompt words through these tasks. When using prefix caching (or reusing key-value (KV) caches), the prompt words are first split into multiple sub-prompt words. These sub-prompt words are then encoded (or vectorized), and position indices and prompt word identifiers are added to obtain multiple first vectors corresponding to each sub-prompt word. Next, based on the position indices of the sub-prompt words, a grouped search is performed in the database to match similar vectors. If similar vectors exist, the prompt word identifier is used to verify whether the matching results come from the same historical prompt word. If a matching result exists, during the full inference phase, the prompt word identifier is used to find the corresponding historical KV cache, and this KV cache is reused for inference. If no historical KV cache exists, a new KV is calculated for inference, and the current full KV is cached. The prompt words for the current model inference task and the newly generated output are segmented, vectorized, and stored in the database with added position indices and prompt word identifiers. The position index and prompt word identifier are used to enable grouped retrieval capabilities, ensuring that the retrieval results come from the same prompt word.

[0120] Figure 4 is just a simple description of the system architecture; a detailed explanation will follow with flowcharts.

[0121] In this system architecture, similar vectors are retrieved from the database using the vectors of the split sub-prompt words, thus reusing the corresponding key-value cache. This not only reduces unnecessary redundant calculations and improves inference efficiency, but also enables fuzzy matching, increasing the reuse rate of historical key-value caches.

[0122] Please refer to Figure 5, a flowchart illustrating a data processing method provided in this application. This method may include steps 501 to 505. Steps 501 to 505 can be executed by a data processing device, or by some components of the data processing device (e.g., a processor, chip, or chip system), or by a logic module or software capable of implementing all or part of the functions of the data processing device. The following description uses execution by a data processing device as an example. The processing performed by a single execution entity in steps 501 to 505 can also be divided into multiple execution entities, which can be logically and / or physically separated. The data processing device can be the data processing device in Figure 3A, or the terminal device in Figure 3B, etc. Of course, this method can also be executed by a system composed of a data processing device and a terminal device (this can also be understood as the terminal device providing user information to the data processing device, and the data processing device executing the method provided in this application based on the user information).

[0123] Due to the long intervals between the steps, steps 501 to 505 are briefly described here first, and then described in detail later. Step 501: Obtain the prompt words for the model inference task. Step 502: Split the prompt words to obtain multiple sub-prompt words. Step 503: Vectorize each sub-prompt word to obtain multiple first vectors corresponding to each sub-prompt word. Step 504: Perform similarity matching in the database based on the multiple first vectors to obtain at least one second vector that matches at least one of the multiple first vectors. Step 505: Determine the processing result of the model inference task based on the KV cache corresponding to each of the at least one second vector. The following is a detailed description of each step:

[0124] Step 501: Obtain the prompts for the model inference task.

[0125] In this application, the data processing device can obtain prompt words in various ways, such as through user input, by receiving messages from other devices, or by selecting from a database, etc. No specific method is limited here.

[0126] In this context, prompts for model inference tasks can be used to instruct the model on the inference task itself. Alternatively, prompts can be understood as requests or instructions directed to the model input, guiding the model to generate corresponding responses or complete the inference task. A model inference task can be understood as a user-input request or the corresponding inference task. Prompts can be user-input requests or combinations of user-input requests with other information (such as preset templates, relevant professional knowledge, or completion requests). Specific details are not limited here.

[0127] For example, if a user enters the request information "How's the weather?", the prompt could be something like "How's the weather?" or "How's the weather today?"

[0128] Optionally, the prompt words can be instructional, question-based, descriptive, etc., without specific limitations here. Additionally, the prompt words can take one or more of the following forms: text, audio, image, video, etc., without specific limitations here. Correspondingly, the model inference task can be image processing (e.g., image generation, image enhancement, image format conversion, etc.), text processing (e.g., text generation, text translation, etc.), speech processing (e.g., speech generation, speech conversion, etc.), etc., without specific limitations here.

[0129] In one possible implementation, the data processing device is a terminal device that obtains prompt words through user input. Alternatively, it obtains prompt words by receiving a model inference task sent by the user and processing the task (e.g., translation, transformation, mapping).

[0130] In another possible implementation, the data processing device is a cloud device, which obtains the prompt words by receiving them from the terminal device. Alternatively, the terminal device forwards the model inference task sent by the user to the cloud device, which processes the model inference task (e.g., translation, transformation, mapping) to obtain the prompt words.

[0131] Step 502: Split the prompt words to obtain multiple sub-prompt words.

[0132] After the data processing device obtains the prompt word, it can split the prompt word into multiple sub-prompt words.

[0133] In this embodiment of the application, there are various rules for splitting prompt words. Prompt words can be split according to a preset size (or preset size), words, phrases, strings, characters, or other splitting rules. No specific rules are limited here.

[0134] The preset size (or preset value) can be the block size of the KV cache, or other values ​​related to memory, etc., without limitation here. In addition, the block size can refer to the size of the physical storage block storing the KV cache, or the size of the virtual storage block corresponding to the physical storage block storing the KV cache, without limitation here.

[0135] Optionally, the data processing device splits the hint word into multiple sub-hint words based on the block size of the KV cache. Alternatively, the number of sub-hint words can be understood as being related to the block size of the KV cache. Or, the number of tokens corresponding to each sub-hint word can be understood as being related to the block size of the KV cache. Or, the number of token identifiers corresponding to each sub-hint word can be understood as being related to the block size of the KV cache.

[0136] Optionally, a block stores the key-value pair (KV) corresponding to a block-size token ID. A space stores the KV pair corresponding to a token ID. A sub-hint is stored within a block. Tokens that do not meet the block size requirement will not be stored.

[0137] In this embodiment, the data processing device can split the prompt words into multiple sub-prompt words based on the block size of the KV cache in several ways, which are described below:

[0138] In the first case, the number of tokens for each sub-prompt obtained from splitting the prompt word is the same as the block size of the KV cache.

[0139] This can also be understood as the number of token identifiers in each of the multiple sub-tips obtained by splitting the hint word being the same as the block size of the KV cache. Or it can be understood as an average split.

[0140] For example, as shown in Figure 6, taking the splitting of a prompt word into 6 sub-prompt words as an example, the 6 sub-prompt words in Figure 6 include: sub-prompt word 1 (split id1), sub-prompt word 2 (split id2), sub-prompt word 3 (split id3), sub-prompt word 4 (split id4), sub-prompt word 5 (split id5), and sub-prompt word 6 (split id6). Furthermore, the number of token IDs for each of the 6 sub-prompt words is the same as the block size. Assuming the block size is 128 bits, the number of token IDs corresponding to the sub-prompt words after being encoded by the tokenizer is 128.

[0141] In the second case, the block size is the difference in the number of tokens corresponding to any two adjacent sub-prompts among the multiple sub-prompts obtained by splitting the prompt words.

[0142] This can also be understood as follows: the difference in the number of tokens between any two adjacent sub-prompts obtained from splitting the prompt word is the block size. Alternatively, it can be understood as the number of token IDs in each sub-prompt word increasing by one block size compared to the previous sub-prompt word. Or, it can be understood as the number of token IDs in each sub-prompt word decreasing by one block size compared to the next sub-prompt word. Or, it can be understood as including splitting.

[0143] For example, as shown in Figure 7, taking the splitting of a prompt word into four sub-prompt words as an example, the four sub-prompt words in Figure 7 include: sub-prompt word 1 (split id1), sub-prompt word 2 (split id2), sub-prompt word 3 (split id3), and sub-prompt word 4 (split id4). Furthermore, the number of token IDs for each sub-prompt word increases by one block size compared to the previous sub-prompt word. Assuming the block size is 128 bits, if the number of token IDs corresponding to sub-prompt word 1 after tokenizer encoding is 128, then the number of token IDs corresponding to sub-prompt word 2 after tokenizer encoding is 128 + 128 = 256.

[0144] Step 503: Vectorize each of the multiple sub-prompt words to obtain multiple first vectors that correspond one-to-one with the multiple sub-prompt words.

[0145] After acquiring multiple sub-prompt words, the data processing device vectorizes each sub-prompt word to obtain multiple first vectors. The number of first vectors is the same as the number of sub-prompt words. Alternatively, it can be understood that there is a one-to-one correspondence between the multiple first vectors and the multiple sub-prompt words. Or, it can be understood that each sub-prompt word, after vectorization, yields one first vector.

[0146] The vectorization processing in this embodiment includes at least semantic encoding. Optionally, the vectorization processing may also include adding one or more of the following to the semantic vector obtained by semantic encoding: prompt word identifier, sub-prompt word identifier, position index, etc., which are not limited here.

[0147] Optionally, one of the multiple sub-suggestions has a position index, which indicates the position of the sub-suggestion among the multiple sub-suggestions.

[0148] The concept of a sub-prompt word having a positional index can be interpreted in several ways. For example, the first vector corresponding to a sub-prompt word may include a semantic vector and a positional index (i.e., the positional index is an element included in the first vector), and the positional index can be either before or after the semantic vector; this is not limited here. Another example is that a sub-prompt word or its corresponding first vector may have an attribute that reflects the positional index (i.e., the positional index is not directly reflected in the elements included in the first vector), etc., which is also not limited here.

[0149] Optionally, multiple sub-prompt words or multiple first vectors may correspond to the same prompt word identifier. This prompt word identifier is used to indicate that multiple sub-prompt words belong to the same prompt word, or to indicate that multiple first vectors are associated with the same prompt word.

[0150] For example, vectorization processing includes semantic encoding, adding a position index, and adding a prompt word identifier. Accordingly, each of the multiple first vectors includes a semantic vector, a prompt word identifier (or first identifier), and a position index. One semantic vector corresponds to one sub-prompt word, or in other words, one semantic vector corresponds to one first vector. The first identifier is used to identify the prompt word, and the position index is used to indicate the position of the sub-prompt word corresponding to the first vector among the multiple sub-prompt words.

[0151] For example, continuing the example from Figure 6, the vectorization processing of the six sub-prompt words shown in Figure 6 is shown in Figure 8. Sub-prompt word 1 (split id1) is processed by a vector encoder to obtain semantic vector 1 (corresponding to vector id1), and a position index and prompt word identifier are added to semantic vector 1 to obtain first vector 1. Similarly, sub-prompt word 2 (split id2) is vectorized to obtain first vector 2. Sub-prompt word 3 (split id3) is vectorized to obtain first vector 3. Sub-prompt word 4 (split id4) is vectorized to obtain first vector 4. Sub-prompt word 5 (split id5) is vectorized to obtain first vector 5. Sub-prompt word 6 (split id6) is vectorized to obtain first vector 6.

[0152] For example, continuing the example from Figure 7, the vectorization processing of the four sub-prompt words shown in Figure 7 is shown in Figure 9. Sub-prompt word 1 (split id1) is processed by a vector encoder to obtain semantic vector 1 (corresponding to vector id1), and a position index and prompt word identifier are added to semantic vector 1 to obtain first vector 1. Similarly, sub-prompt word 2 (split id2) is vectorized to obtain first vector 2. Sub-prompt word 3 (split id3) is vectorized to obtain first vector 3. Sub-prompt word 4 (split id4) is vectorized to obtain first vector 4.

[0153] Step 504: Perform similarity matching in the database based on multiple first vectors to obtain at least one second vector that matches at least one of the multiple first vectors.

[0154] After acquiring multiple first vectors, the data processing device performs similarity matching in a database based on these first vectors to obtain at least one second vector that matches at least one of the first vectors. Alternatively, the data processing device retrieves data from the database based on the positional indices of the multiple first vectors to obtain at least one second vector that matches at least one of the first vectors. These multiple second vectors are stored in a key-value cache. The database stores multiple second vectors, which are similar to the multiple first vectors and will not be elaborated further here.

[0155] Optionally, the second vector corresponding to a KV cache can be understood as the second vector corresponding to a prompt word identifier, which is associated with a KV cache. That is, the data processing device can determine the prompt word identifier corresponding to the second vector and retrieve the corresponding KV cache by looking up the prompt word identifier in a table. The table stores the association between the prompt word identifier and the KV cache.

[0156] Alternatively, it can be understood that data processing devices can perform grouped searches based on location indexes. This search process largely preserves the location information of the sub-tips corresponding to the semantic vectors.

[0157] Optionally, the above retrieval process can also be called fuzzy matching, that is, multiple first vectors correspond one-to-one with multiple second vectors, and the similarity between any pair of corresponding first vectors and second vectors is greater than or equal to a threshold.

[0158] The method for calculating similarity is not limited in this application; it can be cosine similarity, Euclidean distance, Hamming distance, Pearson correlation coefficient, etc. The specific method is not limited here.

[0159] For example, if cosine similarity is used, the closer the cosine of the angle between two vectors is to 1, the more similar the two vectors are. The closer the cosine is to 0, the less similar the two vectors are. As another example, if Euclidean distance is used, the smaller the distance between two vectors, the more similar the two vectors are. The larger the distance between two vectors, the less similar the two vectors are.

[0160] It's understandable that the number of multiple first vectors is greater than or equal to the number of multiple second vectors. Alternatively, it can be understood that if each of the multiple first vectors can retrieve a second vector, and the retrieved second vectors correspond to the same historical suggestion term, then the number of multiple first vectors is the same as the number of multiple second vectors. If there is a first vector among the multiple first vectors that fails to retrieve a second vector, then the number of multiple first vectors is greater than the number of multiple second vectors, meaning that at least one first vector did not find a search result.

[0161] In this application embodiment, there are multiple retrieval methods, which are described below:

[0162] The first scenario is a serial retrieval.

[0163] This can also be understood as the data processing device performing similarity matching in the database sequentially based on the positional indexes of multiple first vectors, obtaining at least one second vector, where each second vector shares the same prompt word identifier. Alternatively, it can be understood as the data processing device sequentially searching the database based on the positional indexes of multiple first vectors to obtain at least one second vector.

[0164] For example, each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs.

[0165] Optionally, the data processing device retrieves similar semantic vectors (i.e., the semantic vector corresponding to the second vector) sequentially from front to back based on the position index of the sub-prompt words. Specifically, during each retrieval process from front to back, the retrieval continues when the similarity threshold is met. When the similarity threshold is not met, the retrieval stops and similar results are returned. Alternatively, it can be understood as retrieving sequentially from front to back based on the position index; if a match is found, the retrieval continues; if no match is found, the retrieval stops.

[0166] Understandably, to ensure the accuracy of reusing the KV cache of similar vectors later, the retrieved similar semantic vectors must correspond to the same historical prompt word. In this retrieval process, the position index of the semantic vector is equivalent to K, and the purpose of the retrieval is to reuse the KV cache of similar vectors. That is, based on the K of the model's inference task, historical similarity K is found, and the corresponding V is reused, thereby achieving the reuse of the historical KV cache.

[0167] For example, as shown in Figure 10, assume that the semantic vectors corresponding to the second vector in the database (i.e., the semantic vectors corresponding to historical prompt words) include: semantic vector a (vector ida), semantic vector b (vector idb), semantic vector c (vector idc), semantic vector d (vector idd), semantic vector e (vector ide), and semantic vector f (vector idf). The semantic vectors in the first vector (i.e., the first vector to be fuzzy matched) corresponding to the model inference task include: semantic vector 1 (vector id1), semantic vector 2 (vector id2), semantic vector 3 (vector id3), semantic vector 4 (vector id4), semantic vector 5 (vector id5), and semantic vector 6 (vector id6).

[0168] It is understood that "abcdef" and "123456" simply represent sorting, i.e., positional indexes. Other positional indexing methods may exist in other embodiments, which are not limited here.

[0169] Specifically, based on semantic vector 1, a similar semantic vector 'a' is found in the database. Then, based on semantic vector 2, a similar semantic vector 'b' is found in the database, and so on. The search stops when no similar semantic vector is found in the database based on semantic vector 5. Alternatively, this can be understood as semantic vectors e and f in Figure 10 being dissimilar to semantic vectors 5 and 6 in the model's inference task. Or, the number of tokenids corresponding to semantic vectors e and f does not meet the size requirement.

[0170] The second scenario is parallel retrieval.

[0171] This can also be understood as the data processing device performing similarity matching from the database based on the position indices of multiple first vectors, obtaining at least one second vector, where each of the at least one second vector shares the same prompt word identifier. Alternatively, it can be understood as the data processing device retrieving multiple second vectors from the database based on the position indices of multiple first vectors.

[0172] For example, each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs.

[0173] Optionally, to ensure the accuracy of subsequent reuse of the KV cache of similar vectors, the retrieved similar semantic vectors must correspond to the same historical prompt word. In this retrieval process, the position index of the semantic vector is equivalent to K, and the purpose of the retrieval is to reuse the KV cache of similar vectors. That is, based on the K of the model's inference task, historical similarity K is found, and the V corresponding to historical similarity K is reused, thereby achieving the reuse of the historical KV cache.

[0174] Alternatively, in this scenario, multiple second vectors can be retrieved from the database based on the positional indices of multiple first vectors. It is then determined whether these multiple second vectors correspond to the same historical suggestion term. If not, the search is interrupted, and the search results are returned.

[0175] For example, as shown in Figure 11, assume that the semantic vectors in the database include: semantic vector a (vector ida), semantic vector b (vector idb), semantic vector c (vector idc), semantic vector d (vector idd), and semantic vector e (vector ide). The semantic vectors in the first vector (i.e., the first vector to be fuzzy matched) corresponding to the model inference task include: semantic vector 1 (vector id1), semantic vector 2 (vector id2), semantic vector 3 (vector id3), semantic vector 4 (vector id4), semantic vector 5 (vector id5), and semantic vector 6 (vector id6).

[0176] Specifically, based on semantic vector 1, a similar semantic vector 'a' is found in the database. Then, based on semantic vector 2, a similar semantic vector 'b' is found in the database, and so on. The search stops when no similar semantic vector is found in the database based on semantic vector 5. Alternatively, this can be understood as: semantic vector 'e' in Figure 11 is not similar to semantic vectors 5 and 6 in the model's inference task; or the number of tokenids corresponding to semantic vector 'e' does not meet the size requirement; or semantic vector 'e' does not belong to the same historical prompt words as the previously fuzzy matched semantic vectors (i.e., semantic vectors a, b, c, and d).

[0177] It is understood that the above search methods are just examples. In other embodiments, there may be other search methods, which are not limited here.

[0178] Step 505: Determine the processing result of the model inference task based on the KV cache corresponding to at least one second vector.

[0179] After the data processing device retrieves at least one corresponding second vector from among multiple first vectors, it determines the processing result of the model inference task based on the KV cache corresponding to at least one second vector.

[0180] Alternatively, this can be understood as the data processing device retrieving a similar second vector from the database based on the first vector of the model inference task, and using the KV cache corresponding to the similar second vector to determine the processing result of the model inference task. Or, it can be understood as the data processing device reusing the KV cache of the similar second vector based on the retrieval. That is, the data processing device can find historical similarity K based on the K of the model inference task, and reuse the V corresponding to the historical similarity K, thereby achieving the reuse of historical KV caches.

[0181] Optionally, for a first vector where the second vector is retrieved, the key-value (KV) cache for the second vector is reused. For a first vector where the second vector is not retrieved, the KV is recalculated. The processing result of the model inference task is determined based on the reused KV cache and the recalculated KV.

[0182] Of course, if the model inference task can retrieve the corresponding second vector, the key-value cache of the second vector can be reused directly without recalculation. Similarly, if the model inference task fails to retrieve the corresponding second vector, an empty result is returned, and recalculation is performed.

[0183] Further optionally, one of the multiple first vectors has a prompt word identifier, which is used to identify the prompt word to which the first vector belongs; the data processing device can also store multiple first vectors in the database based on the prompt word identifier of the first vector, and the multiple first vectors correspond to a KV cache.

[0184] Alternatively, it can be understood that if a match is successful, the historical key-value cache is reused for inference. The newly generated inference results are then segmented, vectorized, and with the addition of position index markers and prompt IDs, stored in the database.

[0185] For example, to facilitate operations such as adding, deleting, and searching for matching vectors, a record stored in the database can include, but is not limited to, the following fields: prompt id (i.e., the input prompt id, used to record which prompts the current vector belongs to), position id (indicating the position index of the current child prompt), vector (indicating the vector value of the current child prompt, i.e., the semantic vector), vector id (indicating the id of the current vector), and count (the default value is 0, indicating the number of times the vector is reused; vector aging is based on this flag).

[0186] In this embodiment, the data processing device vectorizes multiple sub-prompts corresponding to the prompt words and searches for similar vectors, thus enabling the use of the KV cache corresponding to these similar vectors during inference. This reduces unnecessary repetitive calculations and improves inference efficiency. Furthermore, fuzzy matching based on similarity improves the reuse rate of historical KV caches. Additionally, the application provides multiple prompt word splitting methods and multiple retrieval methods, enhancing the flexibility of the data processing approach.

[0187] The data processing method in this application has been described above. The data processing device in this application is described below. Please refer to Figure 12, which shows an embodiment of the data processing device in this application. This data processing device can implement the functions of the data processing device in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this application, the data processing device can be the data processing device in Figure 3A above, or it can be the terminal device in Figure 3B above. The data processing device includes: an acquisition module 1201 and a processing module 1202.

[0188] The acquisition module 1201 is used to acquire prompt words for the model inference task;

[0189] Processing module 1202 is used to split the prompt words into multiple sub-prompt words;

[0190] The processing module 1202 is also used to perform vectorization processing on each of the multiple sub-prompt words to obtain multiple first vectors that correspond one-to-one with the multiple sub-prompt words;

[0191] The processing module 1202 is also used to perform similarity matching in the database based on multiple first vectors to obtain at least one second vector that matches at least one of the multiple first vectors, wherein the similarity between a matching first vector and a second vector is greater than or equal to a threshold, and the database includes multiple second vectors, each of which has its own KV cache.

[0192] The processing module 1202 is also used to determine the processing result of the model inference task based on the KV cache corresponding to at least one second vector.

[0193] Both the acquisition module 1201 and the processing module 1202 can be implemented in software or in hardware. For example, the implementation of the processing module 1202 will be described below. Similarly, the implementation of the acquisition module 1201 can be referenced from the implementation of the processing module 1202.

[0194] As an example of a software functional unit, the processing module 1202 may include code running on a computing instance. The computing instance can be similar to the description of data processing devices and / or terminal devices in Figure 3B above; that is, a computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Related descriptions can be found in the previous descriptions and will not be repeated here.

[0195] Furthermore, as an example of a hardware functional unit, the processing module 1202 may include at least one computing device, such as a server. Alternatively, the processing module 1202 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0196] In one possible implementation, the acquisition module 1201 is further configured to acquire historical prompt words for the historical model reasoning task, the historical prompt words including multiple historical sub-prompt words; the processing module 1202 is further configured to calculate the KV cache corresponding to each of the multiple historical sub-prompt words; the processing module 1202 is further configured to perform vectorization processing on at least one of the multiple historical sub-prompt words to obtain at least one second vector corresponding one-to-one with at least one historical sub-prompt word; the processing module 1202 is further configured to store at least one second vector and the KV cache corresponding to each of the at least one second vector in the database.

[0197] In one possible implementation, the processing module 1202 is specifically used to split the prompt words based on the block size of the KV cache to obtain multiple sub-prompt words.

[0198] In one possible implementation, the number of token identifiers corresponding to each sub-prompt word in the multiple sub-prompt words is the same as the block size.

[0199] In one possible implementation, the difference in the number of tokens corresponding to any two adjacent sub-prompt words is the block size.

[0200] In one possible implementation, one of the multiple sub-prompt words has a position index, which indicates the position of a sub-prompt word among the multiple sub-prompt words.

[0201] In one possible implementation, each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. The processing module 1202 is specifically used to perform similarity matching in the database in sequence based on the position index of multiple first vectors to obtain at least one second vector, and the prompt word identifiers of at least one second vector are the same.

[0202] In one possible implementation, each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. The processing module 1202 is specifically used to perform similarity matching from the database based on the position index of multiple first vectors to obtain at least one second vector, and the prompt word identifiers of at least one second vector are the same.

[0203] In one possible implementation, one of the multiple first vectors has a prompt word identifier, which is used to identify the prompt word to which the first vector belongs; the processing module 1202 is also used to store multiple first vectors in the database based on the prompt word identifier of the first vector, and the multiple first vectors correspond to a KV cache.

[0204] In one possible implementation, the number of multiple first vectors is greater than or equal to the number of at least one second vector.

[0205] In this embodiment, the operations performed by each unit in the data processing device are similar to those described in the embodiments shown in Figures 1 to 11 above, and will not be repeated here.

[0206] In this embodiment, the processing module 1202 performs vectorization processing on multiple sub-prompts corresponding to the prompt words and searches for similar vectors, so that the KV cache corresponding to the similar vectors can be used during inference. On the one hand, it can reduce unnecessary repetitive calculations and improve inference efficiency. On the other hand, fuzzy matching can be achieved based on similarity, improving the reuse rate of historical KV cache. Furthermore, it provides multiple ways to split prompt words and multiple retrieval methods, improving the flexibility of data processing methods.

[0207] Please refer to Figure 13, which is another schematic structural diagram of the data processing device provided in this application. The data processing device includes a logic circuit 1301 and an input / output interface 1302. The data processing device can be an integrated circuit, etc.

[0208] The acquisition module 1201 shown in Figure 12 can be a communication interface, which can be the input / output interface 1302 in Figure 13. The input / output interface 1302 can include an input interface and an output interface. Alternatively, the communication interface can also be a transceiver circuit, which can include an input interface circuit and an output interface circuit. The processing module 1202 shown in Figure 12 can be the logic circuit 1301 in Figure 13.

[0209] The logic circuit 1301 and the input / output interface 1302 can also perform other steps executed by the data processing device in any of the embodiments of Figures 1 to 11 and achieve corresponding beneficial effects, which will not be elaborated here.

[0210] Optionally, the logic circuit 1301 can be a processing device, the functions of which can be partially or entirely implemented in software.

[0211] Optionally, the data processing device may include a memory and a processor, wherein the memory is used to store a computer program, and the processor reads and executes the computer program stored in the memory to perform the corresponding processing and / or steps in any of the method embodiments.

[0212] Optionally, the data processing device may consist only of a processor. A memory for storing computer programs is located outside the processing device, and the processor is connected to the memory via circuitry / wires to read and execute the computer programs stored in the memory. The memory and processor may be integrated together or physically independent of each other.

[0213] Optionally, the data processing device may be one or more chips, or one or more integrated circuits. For example, the processing device may be one or more field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), system-on-chips (SoCs), central processing units (CPUs), network processors (NPs), digital signal processors (DSPs), microcontroller units (MCUs), programmable logic devices (PLDs), or other integrated chips, or any group of the above chips or processors.

[0214] Please refer to Figure 14, which shows the data processing device 1400 involved in the above embodiments provided in this application. The data processing device 1400 may be the data processing device in Figure 3A above, or the terminal device in Figure 3B above, etc.

[0215] As shown in Figure 14, the data processing device 1400 includes a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other via the bus 1402. The data processing device 1400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the data processing device 1400.

[0216] Bus 1402 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 14, but this does not imply that there is only one bus or one type of bus. Bus 1402 can include pathways for transmitting information between various components of data processing device 1400 (e.g., memory 1406, processor 1404, communication interface 1408).

[0217] The processor 1404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0218] The memory 1406 may include volatile memory, such as random access memory (RAM). The processor 1404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0219] The memory 1406 stores executable program code, and the processor 1404 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the aforementioned data processing method. That is, the memory 1406 stores instructions for executing the data processing method.

[0220] The communication interface 1408 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the data processing device 1400 and other devices or communication networks.

[0221] This application also provides a computing device cluster for implementing the functions of the data processing device described above. The computing device cluster includes at least one computing device. This computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0222] Please refer to Figure 15, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 15, the computing device cluster includes at least one data processing device 1400. The memory 1406 of one or more data processing devices 1400 in the computing device cluster may store the same instructions for executing data processing methods.

[0223] In some possible implementations, the memory 1406 of one or more data processing devices 1400 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more data processing devices 1400 can jointly execute instructions for executing data processing methods.

[0224] It should be noted that the memory 1406 in different data processing devices 1400 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 1406 of different data processing devices 1400 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0225] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 16 illustrates one possible implementation. Figure 16 is a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 16, in computing device cluster 1600, two data processing devices 1400A and 1400B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1406 in data processing device 1400A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1406 in data processing device 1400B stores instructions for executing the functions of the processing module.

[0226] It should be understood that the functions of the data processing device 1400A shown in Figure 16 can also be performed by multiple data processing devices 1400. Similarly, the functions of the data processing device 1400B can also be performed by multiple data processing devices 1400.

[0227] Furthermore, the data processing method provided in this application embodiment can also be applied to a cloud system. The cloud system includes a cloud management platform and infrastructure. The cloud management platform is used to manage the infrastructure that provides cloud services. The cloud management platform is used to obtain prompt words for the model inference task and instruct the infrastructure to perform one or more of the following: splitting the prompt words to obtain multiple sub-prompt words; vectorizing each of the multiple sub-prompt words to obtain multiple first vectors that correspond one-to-one with the multiple sub-prompt words; performing similarity matching in the database based on the multiple first vectors to obtain at least one second vector that matches the meaning of at least one of the multiple first vectors; and determining the processing result of the model inference task based on the KV cache corresponding to each of the at least one second vector.

[0228] Alternatively, this application embodiment also provides a cloud system, which includes at least one module shown in FIG12. Alternatively, the cloud system includes at least one data processing device or a component of a data processing device shown in FIG13 to FIG16.

[0229] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0230] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0231] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0232] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions may be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized by, The method includes: Obtain prompts for the model inference task; The prompt word is split into multiple sub-prompt words; Each of the plurality of sub-prompt words is vectorized to obtain a plurality of first vectors that correspond one-to-one with the plurality of sub-prompt words; Based on the plurality of first vectors, similarity matching is performed in the database to obtain at least one second vector that matches at least one of the plurality of first vectors. The similarity between a matching first vector and a second vector is greater than or equal to a threshold. The database includes a plurality of second vectors, and each second vector has its own KV cache. The processing result of the model inference task is determined based on the KV cache corresponding to each of the at least one second vector.

2. The method of claim 1, wherein, The method further includes: Obtain historical prompt words for historical model reasoning tasks, wherein the historical prompt words include multiple historical sub-prompt words; Calculate the KV cache corresponding to each of the multiple historical sub-prompt words; Vectorize at least one of the multiple historical sub-prompt words to obtain at least one second vector that corresponds one-to-one with the at least one historical sub-prompt word; The at least one second vector and the KV cache corresponding to each of the at least one second vector are stored in the database.

3. The method according to claim 1 or 2, characterized in that, The splitting of the prompt word yields multiple sub-prompt words, including: The prompt words are split based on the block size of the KV cache to obtain the multiple sub-prompt words.

4. The method of claim 3, wherein, The number of token identifiers corresponding to each of the multiple sub-prompt words is the same as the block size.

5. The method of claim 3, wherein, The difference in the number of token identifiers corresponding to any two adjacent sub-prompt words among the plurality of sub-prompt words is the block size.

6. The method according to any one of claims 1 to 5, characterized in that, One of the plurality of sub-prompt words has a position index, which indicates the position of the one sub-prompt word among the plurality of sub-prompt words.

7. The method of claim 6, wherein, Each second vector corresponds to a prompt word identifier, which identifies the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. The similarity matching based on the multiple first vectors in the database includes: Based on the position index sorting of the plurality of first vectors, similarity matching is performed sequentially in the database to obtain at least one second vector, wherein the prompt word identifiers of the at least one second vector are the same.

8. The method of claim 6, wherein, Each second vector corresponds to a prompt word identifier, which identifies the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. The similarity matching based on the multiple first vectors in the database includes: Based on the position indexes of the plurality of first vectors, similarity matching is performed from the database to obtain at least one second vector, wherein the prompt word identifiers of the at least one second vector are the same.

9. The method according to any one of claims 1 to 8, characterized in that, One of the plurality of first vectors has a prompt word identifier, which is used to identify the prompt word to which the first vector belongs; The method further includes: The plurality of first vectors are stored in the database based on the prompt word identifiers of the first vectors, and the plurality of first vectors correspond to the KV cache.

10. The method according to any one of claims 1 to 9, characterized in that, The number of the plurality of first vectors is greater than or equal to the number of the at least one second vector.

11. A data processing device, characterized by The data processing device includes: The acquisition module is used to acquire prompt words for model inference tasks; The processing module is used to split the prompt word into multiple sub-prompt words; The processing module is further configured to perform vectorization processing on each of the plurality of sub-prompt words to obtain a plurality of first vectors corresponding one-to-one with the plurality of sub-prompt words; The processing module is further configured to perform similarity matching in the database based on the plurality of first vectors to obtain at least one second vector that matches at least one of the plurality of first vectors, wherein the similarity between a matching first vector and a second vector is greater than or equal to a threshold, and the database includes a plurality of second vectors, each of which has a corresponding KV cache. The processing module is further configured to determine the processing result of the model inference task based on the KV cache corresponding to each of the at least one second vector.

12. The data processing device according to claim 11, characterized in that, The acquisition module is also used to acquire historical prompt words for historical model reasoning tasks, and the historical prompt words include multiple historical sub-prompt words; The processing module is also used to calculate the KV cache corresponding to each of the multiple historical sub-prompt words; The processing module is further configured to perform vectorization processing on at least one of the multiple historical sub-prompt words to obtain at least one second vector corresponding one-to-one with the at least one historical sub-prompt word; The processing module is further configured to store the at least one second vector and the KV cache corresponding to each of the at least one second vector into the database.

13. The data processing device according to claim 11 or 12, characterized in that, The processing module is specifically used to split the prompt words into multiple sub-prompt words based on the block size of the KV cache.

14. The data processing device according to claim 13, characterized in that, The number of token identifiers corresponding to each of the multiple sub-prompt words is the same as the block size.

15. The data processing device according to claim 13, characterized in that, The difference in the number of token identifiers corresponding to any two adjacent sub-prompt words among the plurality of sub-prompt words is the block size.

16. The data processing device according to any one of claims 11 to 15, characterized in that, One of the plurality of sub-prompt words has a position index, which indicates the position of the one sub-prompt word among the plurality of sub-prompt words.

17. The data processing device according to claim 16, characterized in that, Each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. The processing module is specifically used to perform similarity matching in the database in sequence based on the position index sorting of the multiple first vectors to obtain at least one second vector. The prompt word identifiers of the at least one second vector are the same.

18. The data processing device according to claim 16, characterized in that, Each second vector corresponds to a prompt word identifier, which is used to identify the prompt word to which the second vector belongs. The second vector is obtained by vectorizing the sub-prompt words of the prompt word to which the second vector belongs. The processing module is specifically used to perform similarity matching from the database based on the position index of the multiple first vectors to obtain at least one second vector. The prompt word identifiers of the at least one second vector are the same.

19. The data processing device according to any one of claims 11 to 18, characterized in that, One of the plurality of first vectors has a prompt word identifier, which is used to identify the prompt word to which the first vector belongs; The processing module is further configured to store the plurality of first vectors in the database based on the prompt word identifiers of the first vectors, and the plurality of first vectors correspond to the KV cache.

20. The data processing device according to any one of claims 11 to 19, characterized by, The number of the plurality of first vectors is greater than or equal to the number of the at least one second vector.

21. A cluster of computing devices, characterized in that, The computing device cluster includes at least one computing device, each computing device including a processor and memory: The memory is used to store instructions; The processor is configured to, according to the instructions, cause the computing device cluster to perform the method of any one of claims 1 to 10.

22. A computer storage medium, comprising, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 10.

23. A computer program product, characterised in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 10.