Pre-filling method and related apparatus
Patent Information
- Application Number
- PCT/CN2026/090318
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-04-18
- Filing Date
- 2026-04-13
- Publication Date
- 2026-10-22
Smart Images
Figure CN2026090318_22102026_PF_FP_ABST
Abstract
Description
Pre-filling method and related apparatus
[0001] This application claims priority to Chinese Patent Application No. 202510491564.8, filed on April 18, 2025, entitled "Prefilling Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of communication technology, and in particular to a pre-filling method and related apparatus. Background Technology
[0003] Large language models (LLMs) are artificial intelligence models based on deep learning technology. Through training on massive amounts of text data, they master language rules and perform tasks such as text generation, understanding, and reasoning. Retrieval-augmented generation (RAG) is a technique that leverages information from specific relevant data sources to enhance the accuracy and reliability of generative models. In the pre-filling stage, RAG technology can retrieve external knowledge bases (such as vector databases) and concatenate relevant document fragments into the user's original input, thereby optimizing the generation effect of LLM. When determining relevant document fragments requested by the user based on RAG technology, key-value caching techniques can be used to reuse cached key-value pairs to reduce computational load.
[0004] However, existing key-value caching applications in RAG technology have low key-value cache hit rates, especially for long sequences, resulting in minimal acceleration of the Prefill stage. Summary of the Invention
[0005] This application provides a pre-filling method and related apparatus to solve the technical problem that key-value caching technology has a relatively small acceleration effect on the pre-filling stage.
[0006] In a first aspect, this application provides a pre-filling method, comprising: performing a retrieval based on a user's query request to obtain retrieval results, the retrieval results including multiple text chunks; determining key-value pairs corresponding to each chunk in the retrieval results based on a first key-value cache, the first key-value cache including key-value pairs of multiple chunks; and determining a second key-value cache corresponding to the retrieval results based on the key-value pairs corresponding to each chunk in the retrieval results.
[0007] Thus, based on the user's query request, a retrieval is performed to obtain the retrieval results, which include multiple chunks. Then, the key-value pairs of each chunk in the retrieval results are retrieved based on the first key-value cache, which also includes key-value pairs from multiple chunks. Based on the key-value pairs of the multiple chunks in the retrieval results, the corresponding second key-value cache is determined. The pre-filling process determines the key-value cache key-value pairs corresponding to each chunk at the chunk granularity. The pre-stored key-value pairs are also stored at the chunk granularity. Compared to determining key-value pairs for long texts, the chunk granularity is smaller, reducing the matching granularity and making it easier to retrieve the key-value pairs of each chunk in the retrieval results from the pre-stored first key-value cache. This improves the hit rate of the first key-value cache, enhances its benefits, and accelerates the pre-filling process.
[0008] Optionally, the electronic device has a preset access record, which is used to indicate the number of times each chunk is accessed. After the retrieval is performed based on the user's query request and the retrieval result is obtained, the method further includes: updating the access record based on the retrieval result, which is used to indicate the number of times each chunk is accessed; and updating the first key-value cache based on the updated access record, wherein the number of times the chunk corresponding to the key-value pair in the first key-value cache is accessed is greater than the number of times other chunks are accessed.
[0009] In this way, by updating the key-value pairs in the first key-value cache in real time through access records, the quality of the key-value pairs in the first key-value cache is improved (e.g., the number of accesses is high), thereby improving the hit rate of the first key-value cache and thus improving the acceleration effect of the key-value pair pre-filling stage.
[0010] Furthermore, if multiple key-value pairs in the first key-value cache all correspond to chunks that meet preset conditions, the number of key-value pairs in the first key-value cache is limited, preventing an excessive number of key-value pairs from consuming too much storage resources. Moreover, the preset conditions also filter the key-value pairs in the first key-value cache, improving their quality (e.g., higher access frequency), thereby increasing the cache's hit rate.
[0011] The preset condition can be that the number of accesses to the chunk corresponding to the key-value pair is greater than or equal to a preset threshold.
[0012] Optionally, determining the second key-value cache corresponding to the search result based on the key-value pairs corresponding to each chunk in the search result includes: determining the association relationship between the key-value pairs corresponding to multiple chunks in the search result; and determining the second key-value cache corresponding to the search result based on the key-value pairs corresponding to each chunk in the search result and the association relationship.
[0013] Thus, this application transforms the process of obtaining the key-value cache of the search results into the process of obtaining key-value pairs of multiple chunks in the search results. By reducing the matching granularity, the hit rate of the key-value cache is improved. However, the segmentation process results in multiple chunks being independent of each other. The relationship between the key-value pairs of multiple chunks in the search results is determined through association, and based on this relationship, multiple chunks are merged into the key-value cache corresponding to the search results to improve the semantic coherence of multiple key-value pairs in the second key-value cache.
[0014] Optionally, the first key-value cache includes a first group and a second group. The first group includes key-value pairs of at least one chunk, and the second group includes key-value pairs of at least one chunk. The key-value pairs of the first group are stored in a first storage medium, and the key-value pairs of the second group are stored in a second storage medium. The number of accesses to the chunk corresponding to the key-value pairs in the first group is greater than the number of accesses to the chunk corresponding to the key-value pairs in the second group. The access speed of the first storage medium is faster than the access speed of the second storage medium.
[0015] In this way, the key-value pairs of chunks accessed more frequently are stored in storage media with higher access speeds, while the key-value pairs of chunks accessed less frequently are stored in storage media with lower access speeds. The storage location of the key-value pairs corresponding to a chunk is determined by the number of times a chunk is accessed, thereby improving the utilization rate of storage resources and the hit rate of cache resources.
[0016] Optionally, determining the key-value pair corresponding to each chunk in the search result based on the first key-value cache includes: if the chunk in the search result matches the first key-value cache, then obtaining the key-value pair corresponding to the chunk from the first key-value cache; if the chunk in the search result does not match the first key-value cache, then calculating the key-value pair of the chunk.
[0017] Optionally, the method further includes: if the chunk in the search result does not hit the first key-value cache and the chunk meets a preset condition, then the key-value of the chunk is stored in the cache to update the first key-value cache, that is, the key-value pair corresponding to the chunk is stored in the cache to update the first key-value cache.
[0018] Optionally, the retrieval result includes a first chunk and a second chunk, where the first chunk hits the first key-value cache and the second chunk does not hit the first key-value cache, or in other words, the first key-value cache includes the key-value pair corresponding to the first chunk but does not include the key-value pair corresponding to the second chunk. In this case, the key-value pair corresponding to the first chunk is retrieved from the first key-value cache and the key-value pair corresponding to the second chunk is calculated in parallel.
[0019] In a second aspect, a prefilling apparatus is provided, the apparatus comprising a unit for performing the prefilling method described in any one aspect.
[0020] Thirdly, an electronic device is provided, comprising: a memory including computer-readable instructions; and a processor communicating with the memory, the processor being configured to execute the computer-readable instructions, causing the electronic device to perform the pre-filling method described in any one of the first aspects.
[0021] Fourthly, a computer-readable storage medium is provided, including a program or instructions that, when executed by a processor, implement the pre-filling method as described in any one of the first aspects.
[0022] Fifthly, a chip is provided, including a processor for retrieving and executing instructions stored in a memory, causing an electronic device on which the chip is mounted to perform the pre-filling method described in any one of the first aspects.
[0023] In a sixth aspect, a computer program product is provided, the computer program product comprising instructions that, when executed by a computer, implement the method as described in any one of the first aspects.
[0024] The beneficial effects of each possible implementation of the pre-filling device provided in the second aspect of the embodiments of this application, the electronic device provided in the third aspect, the computer-readable storage medium provided in the fourth aspect, the chip provided in the fifth aspect, and the computer program product provided in the sixth aspect can be referred to the descriptions of the various possible implementations in the first aspect, and will not be repeated here. Attached Figure Description
[0025] Figure 1 is a schematic diagram of a pre-LLM processing procedure provided in an embodiment of this application;
[0026] Figure 2 is a schematic diagram of the relationship between key-value caches provided in an embodiment of this application;
[0027] Figure 3 is a flowchart illustrating a pre-filling method provided in an embodiment of this application;
[0028] Figure 4 is a schematic diagram of a pre-filling process provided in an embodiment of this application;
[0029] Figure 5 is a schematic diagram of a pre-filling device provided in an embodiment of this application;
[0030] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0031] The technical solutions in this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of them.
[0032] Large Language Models (LLMs) are artificial intelligence models based on deep learning technology. Through training on massive amounts of text data, they master language rules and perform tasks such as text generation, understanding, and reasoning. As shown in Figure 1, the process of an LLM handling user questions and generating answers can be divided into two main stages: Prefill and Decode. The Prefill stage processes user input requests all at once, generating a key-value cache (KV cache) needed for subsequent decoding. The Decode stage, based on the KV cache generated in the Prefill stage, regressively generates answers chunk by chunk until a termination condition is met. A chunk is obtained by dividing a document into paragraphs, fixed-length sections, or logical units.
[0033] In the Prefill stage, RAG technology can be used to retrieve relevant text from external knowledge bases. This text is then concatenated with the user request and used as input for the Decode stage to optimize the LLM generation. However, when the retrieved content is long and / or the external knowledge base is large, the Prefill stage can become excessively long.
[0034] Key-value caching is one of the core technologies for optimizing inference in large language models within the Transformer architecture. By caching the key and value matrices in the attention mechanism, it significantly reduces redundant computations and improves inference efficiency, thereby reducing the duration of the prefill stage. However, existing key-value caching technologies suffer from low hit rates and have limited acceleration effects on the prefill stage.
[0035] To address the aforementioned issues, this application provides a pre-filling method. Based on a user's query request, a retrieval is performed to obtain retrieval results, which include multiple text chunks. Then, key-value pairs for each chunk in the retrieval results are retrieved based on a first key-value cache, which includes key-value pairs for multiple chunks. A second key-value cache corresponding to the retrieval results is determined based on the key-value pairs for the multiple chunks in the retrieval results. The pre-filling process determines the key-value pairs corresponding to each chunk. Compared to determining key-value pairs for long texts or directly retrieving the key-value cache of the retrieval results, the granularity of the key-value pairs is smaller, reducing the matching granularity of the key-value cache. This makes it easier to retrieve the key-value pairs for each chunk in the retrieval results from pre-stored key-value pairs, thereby improving the hit rate of the key-value cache, increasing the benefits of the key-value cache, and accelerating the pre-filling process.
[0036] Optionally, embodiments of this application can be applied to the pre-filling stage of a large language model. By using key-value caching technology to obtain key-value pairs corresponding to each chunk in the search results, the hit rate of the pre-stored key-value cache can be improved, thus enhancing the acceleration effect of the pre-filling stage of the large language model. Specifically, in the process of determining the prompt for the query request using RAG technology, if the key-value cache related to the user's query request is obtained as the prompt and input into the large language model, the response information corresponding to the query request can be obtained. Embodiments of this application can be applied to the process of determining the key-value cache corresponding to the search results based on RAG.
[0037] The search enhancement generation method proposed in this application relies on an electronic device, which can be a smart device that integrates at least the corresponding reply function, or it can be an application on a smart device. The aforementioned smart device can be a mobile phone, tablet computer, or personal computer, etc.
[0038] Please refer to Figure 2, which is a schematic diagram of the relationship between key-value caches provided in an embodiment of this application. The first key-value cache includes multiple key-value pairs. As shown in Figure 2, the first key-value cache includes key-value pairs of N chunks, where N is a positive integer. Each chunk corresponds to one key-value pair, such as chunk1 corresponding to key-value pair [1], and chunkN corresponding to key-value pair [N]. The first key-value cache can be pre-stored key-value pairs. For example, the key-value pairs of multiple chunks can be pre-calculated and stored in the cache.
[0039] In the Prefill stage, in order to obtain the key-value pair of chunkM, it can be determined whether the first key-value cache contains the key-value pair of chunkM; if the first key-value cache contains the key-value pair of chunkM, then the key-value pair of chunkM is obtained from the first key-value cache; if the first key-value cache does not contain the key-value pair of chunkM, then the key-value pair of chunkM is calculated.
[0040] To obtain the key-value pairs of the search results, we can first obtain the key-value pairs of each chunk in the search results. In Figure 2, the search results include chunk1 to chunk50 and chunk[N+1]. The first key-value cache includes the key-value pairs of chunk1 to chunk50, so the key-value pairs of chunk1 to chunk50 in the first key-value cache can be reused. The first key-value cache does not include the key-value pairs of chunk[N+1], so the key-value pairs of chunk[N+1] are calculated.
[0041] After obtaining the key-value pairs of each chunk in the search results, the second key-value cache corresponding to the search results can be determined based on the key-value pairs of each chunk in the search results. For example, the second key-value cache can be obtained by concatenating the key-value pairs of multiple chunks in the search results and storing them.
[0042] Please refer to Figure 3, which is a flowchart of a pre-filling method provided in an embodiment of this application. The following description takes the application of the pre-filling method to an electronic device as an example. The pre-filling method in Figure 3 includes the following steps: S301 to S303.
[0043] S301. The electronic device performs a search based on the user's query request and obtains search results, which include multiple chunks.
[0044] Optionally, the electronic device retrieves a knowledge base based on the user's query request, and obtains multiple chunks from the knowledge base that are similar to the query request. For example, the similarity between the query request and the chunks in the knowledge base is calculated through semantic similarity to filter multiple chunks with high similarity to the query request from the knowledge base. The knowledge base can be an external knowledge base connected to the electronic device.
[0045] S302. The electronic device determines the key-value pair corresponding to each chunk in the search results based on the first key-value cache, and the first key-value cache includes key-value pairs of multiple chunks.
[0046] Optionally, if the first key-value cache matches key-value pairs corresponding to one or more chunks in the search results, or in other words, the first key-value cache includes key-value pairs corresponding to one or more chunks in the search results, then the key-value pair of the corresponding chunk is retrieved from the first key-value cache. For example, the first key-value cache includes key-value pairs for 100 chunks, with the key-value pairs for the 100 chunks positioned from position 1 to position 100. If chunk A in the search results matches the key-value pair at position 18 in the first key-value cache, then the key-value pair at position 18 is retrieved from the first key-value cache and used as the key-value pair for chunk A.
[0047] In one example, if the first key-value cache does not find a key-value pair corresponding to a certain chunk in the search results, or in other words, the first key-value cache does not contain a key-value pair corresponding to that chunk, then the key-value pair for that chunk is calculated.
[0048] Optionally, since the search results may include multiple chunks, if M chunks among the multiple chunks hit the first key-value cache, then the key-value pairs corresponding to the M chunks are retrieved from the first key-value cache; if N chunks among the multiple chunks do not hit the first key-value cache, then the key-value pairs corresponding to the N chunks are calculated, where M and N are positive integers. In this embodiment, the steps of retrieving the key-value pairs of the M chunks from the first key-value cache and calculating the key-value pairs of the N chunks can be executed in parallel. By asynchronously executing the reading of the key-value cache and the calculation of the key-value pairs, the efficiency of determining multiple chunks in the search results can be improved. Of course, if none of the multiple chunks in the search results hit the first key-value cache, then the key-value pairs of multiple chunks can be calculated in parallel to improve the efficiency of determining the key-value pairs of the chunks.
[0049] Optionally, the electronic device may pre-store access records, which indicate the number of times each chunk is accessed. If the electronic device performs searches through an external knowledge base, the access records can be search records or the number of times each chunk in the external knowledge base is searched. For example, an access counter can be created for each chunk in the knowledge base to maintain the number of times each chunk is accessed. The access records can be stored in the form of a queue. The number of times each chunk is accessed indicates the access popularity of each chunk in the knowledge base.
[0050] One example involves retrieving a user's query request and obtaining search results. These results include multiple chunks related to the query request, i.e., multiple chunks accessed in this instance. The access record can then be updated based on the search results, such as incrementing the access count of each chunk in the access record by 1.
[0051] In one example, the first key-value cache includes key-value pairs of multiple pre-stored chunks. To improve the hit rate of the first key-value cache, it can be updated with updated access records so that the chunk corresponding to the key-value pair in the first key-value cache is accessed more often than other chunks, thereby improving the hit rate of key-value pairs in the first key-value cache and thus improving the reuse rate of key-value pairs.
[0052] In one example, multiple key-value pairs in the first key-value cache correspond to chunks that meet preset conditions. That is, the first key-value cache stores key-value pairs corresponding to chunks that meet the preset conditions. By limiting multiple key-value pairs in the first key-value cache to meet the preset conditions, the number of key-value pairs in the first key-value cache is limited, preventing an excessive number of key-value pairs from consuming too much storage resources. Furthermore, the preset conditions filter the key-value pairs in the first key-value cache, improving their quality (e.g., high access frequency), thereby increasing the cache's hit rate.
[0053] The preset condition can be that the access count of the chunk corresponding to the key-value pair is greater than or equal to a preset threshold. For example, a chunk with an access count greater than 1000 meets the preset condition. Alternatively, the preset condition can be that the chunk corresponding to the key-value pair is among the top N chunks in terms of access count, or that the chunk corresponding to the key-value pair is among a specific percentage of the chunks in terms of access count. For example, the top 100 chunks in terms of access count meet the preset condition, or the top 10% of the chunks in terms of access count meet the preset condition.
[0054] In one example, a chunk that meets a preset condition can be determined based on the updated access record. If the key-value pair of the chunk is not in the first key-value cache, the key-value pair of the chunk can be calculated, and the first key-value cache can be updated based on the key-value pair of the chunk. For example, the key-value pair corresponding to the chunk can be stored in the cache to update the first key-value cache.
[0055] In one example, chunks that do not meet the preset conditions can also be determined based on the updated access records. If the chunk corresponding to the key-value pair in the first key-value cache does not meet the preset conditions, then the key-value pair of that chunk is deleted from the first key-value cache.
[0056] Optionally, the first key-value cache includes key-value pairs of multiple chunks, each chunk having a corresponding number of accesses. For example, chunk1 is accessed 98 times, and chunk2 is accessed 298 times. Different access counts indicate different access frequencies for the chunks. In this embodiment, the key-value pairs of the chunks included in the first key-value cache can be stored hierarchically according to the access frequency of the chunks. For example, key-value pairs corresponding to chunks with high access frequency are stored in storage media with higher access rates, while chunks with low access frequency are stored in storage media with lower access rates.
[0057] In one example, the first key-value cache includes a first group and a second group. The first group includes key-value pairs of at least one chunk, and the second group includes key-value pairs of at least one chunk. The key-value pairs of the first group are stored in a first storage medium, and the key-value pairs of the second group are stored in a second storage medium. The chunks corresponding to the key-value pairs in the first group are accessed more frequently than the chunks corresponding to the key-value pairs in the second group. The access speed of the first storage medium is faster than the access speed of the second storage medium. That is, the key-value pairs of chunks with more frequent accesses are stored in the storage medium with higher access speed, and the key-value pairs of chunks with fewer frequent accesses are stored in the storage medium with lower access speed. The storage location of the key-value pairs corresponding to a chunk is determined by the number of accesses to that chunk, thereby improving the utilization rate of storage resources and the hit rate of cache resources.
[0058] For example, key-value pairs of chunks with access counts in the first range are stored in storage medium 1, key-value pairs of chunks with access counts in the second range are cached in storage medium 2, and key-value pairs of chunks with access counts in the third range are stored in storage medium 3. The values in the first range are greater than the values in the second range, the values in the second range are greater than the values in the third range, the data access rate in storage medium 1 is greater than the data access rate in storage medium 2, and the data access rate in storage medium 2 is greater than the data access rate in storage medium 3. Storage medium 1 is High Bandwidth Memory (HBM), storage medium 2 is Dynamic Random Access Memory (DRAM), and storage medium 3 is a Solid State Disk (SSD).
[0059] S303. The electronic device determines the second key-value cache corresponding to the search result based on the key-value pairs corresponding to each chunk in the search result.
[0060] Optionally, after determining the key-value pairs of each chunk in the search results, multiple key-value pairs can be concatenated to obtain the second key-value cache corresponding to the search results.
[0061] The LLM process can include two phases: the Prefill phase and the Decode phase. The input of the Prefill phase is the query request, and the output of the Prefill phase includes the second key-value cache corresponding to the search results. The input of the Decode phase includes the second key-value cache corresponding to the search results, and the output of the Decode phase includes the response information corresponding to the query request.
[0062] Optionally, after determining the second key-value cache corresponding to the search result, the second key-value cache and the key-value cache corresponding to the query request can be concatenated to obtain the input information of the LLM.
[0063] Optionally, to improve the hit rate of key-value pairs in the first key-value cache, the process of retrieving the key-value cache of the search results is split into the process of retrieving key-value pairs of multiple chunks in the search results. Furthermore, the process of retrieving key-value pairs for each chunk is performed independently. The association relationships between the key-value pairs of multiple chunks can also be obtained, so that the key-value pairs of multiple chunks can be concatenated based on these association relationships to obtain the second key-value cache corresponding to the search results, thereby improving the semantic coherence and computational efficiency of multiple key-value pairs in the second key-value cache.
[0064] In one example, after determining the association between key-value pairs of multiple chunks in the search results, the association and a second key-value cache are also used as input information into the LLM to improve the accuracy of the LLM's response information.
[0065] The association between key-value pairs can be a statistical or semantic connection based on the key, value, or context. Alternatively, in attention mechanisms, it can refer to the semantic interaction between queries (Q) and keys (K) at different layers. Cross attention is a mechanism for calculating the association between different key-value pairs. It is similar to self-attention but applied to different sequences or sets. In key-value caching, cross attention can be used to calculate the association between different key-value pairs.
[0066] Thus, based on the user's query request, a retrieval is performed to obtain the retrieval results, which include multiple chunks. Then, based on the first key-value cache, the key-value pairs of each chunk in the retrieval results are retrieved. The first key-value cache includes key-value pairs of multiple chunks. Based on the key-value pairs of multiple chunks in the retrieval results, the corresponding second key-value cache is determined. The pre-filling process determines the key-value cache key-value pairs corresponding to each chunk at the chunk granularity. The pre-stored key-value pairs are also stored at the chunk granularity. Compared to determining key-value pairs for long texts, the chunk granularity is smaller, reducing the matching granularity and making it easier to retrieve the key-value pairs of each chunk in the retrieval results from the pre-stored first key-value cache. This improves the hit rate of the first key-value cache, enhances its benefits, and accelerates the pre-filling process.
[0067] Referring to Figure 4, which is a schematic diagram of a pre-filling process according to an embodiment of this application, after the electronic device obtains the query request input by the user, it performs a search query through a search engine to obtain the search results. The search results include multiple chunks. The electronic device then queries the first key-value cache to see if it contains the key-value pair corresponding to each chunk in the search results. If the first key-value cache matches the key-value pair corresponding to a chunk in the search results, the key-value pair for that chunk is retrieved from the first key-value cache. If the first key-value cache does not match the key-value pair corresponding to a chunk in the search results, the key-value pair for that chunk is calculated. To improve efficiency, the calculation of the key-value pair and the retrieval of the key-value pair from the first key-value cache can be performed in parallel. Then, the association between the calculated key-value pair and the key-value pair of the chunk retrieved from the first key-value cache is determined. Based on the association, the calculated key-value pair and the key-value pair of the chunk retrieved from the first key-value cache are concatenated to obtain the second key-value cache. The second key-value cache is used as the input to the decode phase of the LLM. The decode phase of the LLM outputs the response information corresponding to the query request.
[0068] After retrieving the query request and obtaining the search results, the system also updates the access records based on the query results. If the access records determine that the chunk corresponding to the calculated key-value pair meets the preset conditions, the key-value pair of that chunk is stored in the cache, and the first key-value cache is updated after a specific time. If the access records determine that the chunk corresponding to the calculated key-value pair does not meet the preset conditions, the key-value pair of that chunk is deleted.
[0069] Please refer to Figure 5, which is a schematic diagram of a pre-filling device provided in an embodiment of this application. This model runtime overhead acquisition device can be deployed in the electronic device described in the foregoing embodiments. As shown in Figure 5, the pre-filling device includes a retrieval unit and a determination unit.
[0070] The retrieval unit is used to retrieve results based on the user's query request. The retrieval results include multiple text chunks.
[0071] The determining unit is used to determine the key-value pair corresponding to each chunk in the search results based on the first key-value cache, which includes key-value pairs of multiple chunks;
[0072] The second key-value cache corresponding to the search results is determined based on the key-value pairs corresponding to each chunk in the search results.
[0073] Optionally, the device pre-stores access records, which are used to indicate the number of times each chunk is accessed. The pre-filling device also includes an update unit, which is used to update the access records based on the search results; and to update a first key-value cache based on the updated access records, wherein the number of accesses of the chunk corresponding to the key-value pair in the first key-value cache is greater than the number of accesses of other chunks.
[0074] Optionally, the determining unit is also used to determine the association relationship between key-value pairs corresponding to multiple chunks in the search results; and to determine the second key-value cache corresponding to the search results based on the key-value pairs and association relationships corresponding to each chunk in the search results.
[0075] Optionally, the first key-value cache includes a first group and a second group. The first group includes key-value pairs of at least one chunk, and the second group includes key-value pairs of at least one chunk. The key-value pairs of the first group are stored in a first storage medium, and the key-value pairs of the second group are stored in a second storage medium. The number of accesses to the chunks corresponding to the key-value pairs in the first group is greater than the number of accesses to the chunks corresponding to the key-value pairs in the second group. The access speed of the first storage medium is faster than the access speed of the second storage medium.
[0076] Optionally, the key-value pairs corresponding to each chunk in the search results are determined based on the first key-value cache, including:
[0077] If the first key-value cache includes the key-value pair corresponding to the chunk in the search result, then retrieve the key-value pair corresponding to the chunk from the first key-value cache;
[0078] If the first key-value cache does not include the key-value pair corresponding to the chunk in the search results, then the key-value pair of the chunk is calculated.
[0079] Optionally, the update unit is also used to: if the chunk in the search results does not hit the first key-value cache and the chunk meets the preset conditions, then store the key-value of the chunk into the cache to update the first key-value cache.
[0080] Optionally, if the retrieval result includes a first chunk and a second chunk, and includes key-value pairs corresponding to the first chunk but does not include key-value pairs corresponding to the second chunk, then the operations of retrieving key-value pairs corresponding to the first chunk from the first key-value cache and calculating key-value pairs corresponding to the second chunk are performed in parallel.
[0081] It should be understood that the above description is merely to help those skilled in the art better understand the embodiments of this application, and is not intended to limit the scope of the embodiments of this application. Based on the examples given above, those skilled in the art can obviously make various equivalent modifications or changes. For example, some steps in the various methods described above may be unnecessary, or new steps may be added. Alternatively, any combination of two or more of the above embodiments may be used. Such modifications, changes, or combinations also fall within the scope of the embodiments of this application.
[0082] It should also be understood that the methods, situations, categories, and classifications of embodiments in this application are for the convenience of description only and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined without contradiction.
[0083] It should also be understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its task and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0084] It should also be understood that the above description of the embodiments of this application focuses on highlighting the differences between the various embodiments. Any similarities or differences not mentioned can be referred to each other. For the sake of brevity, they will not be repeated here.
[0085] The embodiments of the methods and apparatus provided in this application have been described above in conjunction with Figures 1-5. The electronic devices provided in this application are described below.
[0086] This embodiment can divide the electronic device into functional units according to the above method. For example, each function can be divided into different task units, or two or more tasks can be integrated into one processing unit. The integrated unit can be implemented in hardware. It should be noted that the unit division in this embodiment is illustrative and only represents one logical task division; other division methods may be used in actual implementation.
[0087] It should be noted that the relevant content of each step involved in the above method embodiments can be referenced from the task description of the corresponding task unit, and will not be repeated here.
[0088] The electronic device provided in this application embodiment is used to execute the pre-filling method provided in the above method embodiment, and thus can achieve the same effect as the above implementation method.
[0089] In other embodiments, when using integrated units, the electronic device may include a processing unit, a storage unit, and a communication unit. The processing unit can be used to control and manage the actions of the electronic device. For example, it can be used to support the electronic device in executing steps performed by the processing unit. The storage unit can be used to store program code and data, etc. The communication unit can be used to support communication between the electronic device and other electronic devices.
[0090] The processing unit can be a processor or a controller. It can be a set of various exemplary logic blocks, units, and circuits that implement or execute the descriptions in conjunction with the disclosure of this application. The processor can also be a combination of components that perform computational tasks, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage unit can be a memory. The communication unit can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or a device that interacts with other electronic devices or electronic devices.
[0091] Based on the same concept, this application also provides an electronic device. Referring to FIG6, FIG6 shows a schematic diagram of the structure of an exemplary electronic device of this application. The electronic device shown in FIG6 can perform the steps in the pre-filling method performed by any electronic device provided in this application.
[0092] The electronic device 600 includes at least one processor 601, a memory 603, and at least one network interface 604.
[0093] Processor 601 may be, for example, a general-purpose CPU, a digital signal processor (DSP), a network processor (NP), a GPU, a neural network processing unit (NPU), a data processing unit (DPU) for deploying decoy files, a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs) used to implement the scheme of this application, a programmable logic device (PLD), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A PLD may be, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can implement or execute the various logic blocks, modules, and circuits described in connection with the disclosure of this application. A processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0094] Optionally, the electronic device 600 also includes a bus 602. The bus 602 is used to transmit information between the components of the electronic device 600. The bus 602 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 602 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 5, but this does not indicate that there is only one bus or one type of bus.
[0095] Memory 603 may be, for example, read-only memory (ROM) or other types of storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 603 may exist independently and be connected to processor 601 via bus 602. Memory 603 may also be integrated with processor 601.
[0096] Network interface 604 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), or wireless local area network (WLAN). Network interface 604 can include wired network interfaces and wireless network interfaces. Specifically, network interface 604 can be an Ethernet interface, such as Fast Ethernet (FE), Gigabit Ethernet (GE), Asynchronous Transfer Mode (ATM), WLAN, cellular network, or combinations thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In some embodiments of this application, network interface 604 can be used by electronic device 600 to communicate with other devices.
[0097] In specific implementations, as some embodiments, processor 601 may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0098] In specific implementations, as some embodiments, electronic device 600 may include multiple processors. Each of these processors may be a single-core processor or a multi-core processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).
[0099] In some embodiments, memory 603 is used to store program instructions for executing the present application's solution, and processor 601 can execute the program instructions stored in memory 603. That is, electronic device 600 can implement the method provided in the above-described embodiments through processor 601 and the program instructions in memory 603. The program instructions may include one or more software modules. Optionally, processor 601 itself may also store program instructions for executing the present application's solution.
[0100] In specific implementation, the processor 601 in the electronic device 600 of this application reads the instructions in the memory 603, enabling the electronic device 600 shown in FIG6 to execute all or part of the steps in the pre-filling method performed by the electronic device in the above embodiments.
[0101] In the above embodiments, each step of the method is implemented through integrated logic circuits in the hardware of the processor of the electronic device 600 or through software instructions. The steps of the method embodiments disclosed in this application can be directly implemented by the hardware processor, or implemented by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. Since the storage medium is located in memory, the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method embodiments; to avoid repetition, these will not be described in detail here.
[0102] It should be understood that the aforementioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (RISC) machine (ARM) architecture.
[0103] Furthermore, in an alternative embodiment, the memory described above may include read-only memory and random access memory, and provide instructions and data to the processor. The memory may also include non-volatile random access memory. For example, the memory may also store device type information.
[0104] The memory can be volatile or non-volatile, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).
[0105] The electronic device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so they will not be described again here.
[0106] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the methods described in the above-described method embodiments.
[0107] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to implement the method described in the above-described method embodiments.
[0108] This application provides a chip, including a processor, for calling and executing instructions stored in a memory, causing a communication device with the chip installed to execute the method described in the above-described method embodiments of any electronic device provided in this application.
[0109] This application also provides a chip system including a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the method described in the above-described method embodiments. The chip system may be a single chip or a chip module composed of multiple chips.
[0110] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, or magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0111] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium can include various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.
[0112] The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.
[0113] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0114] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0115] It should be understood that in the description of this application and the appended claims, the terms "comprising," "including," "having," and any variations thereof are intended to cover a non-exclusive inclusion and mean "including but not limited to," unless otherwise specifically emphasized. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product, or apparatus.
[0116] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is used to describe the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural.
[0117] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0118] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0119] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein; features defined as "first" or "second" may explicitly or implicitly include at least one of those features.
[0120] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.
[0121] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A pre-filling method, characterized in that, Applied to electronic devices, the method includes: Based on the user's query request, a search is performed to obtain search results, which include multiple text chunks. The key-value pair corresponding to each chunk in the search results is determined based on the first key-value cache, wherein the first key-value cache includes key-value pairs of multiple chunks; The second key-value cache corresponding to the search result is determined based on the key-value pair corresponding to each chunk in the search result.
2. The method according to claim 1, characterized in that, The electronic device has access records pre-stored, which indicate the number of times each chunk is accessed. After retrieving results based on the user's query request, the method further includes: Update the access record based on the search results; The first key-value cache is updated based on the updated access records, wherein the access count of the chunk corresponding to the key-value pair in the first key-value cache is greater than the access count of other chunks.
3. The method according to claim 1 or 2, characterized in that, The step of determining the second key-value cache corresponding to the search result based on the key-value pairs corresponding to each chunk in the search result includes: Determine the association between key-value pairs corresponding to multiple chunks in the search results; The second key-value cache is determined based on the key-value pairs corresponding to each chunk in the search results and the association relationship.
4. The method according to any one of claims 1 to 3, characterized in that, The first key-value cache includes a first group and a second group. The first group includes key-value pairs of at least one chunk, and the second group includes key-value pairs of at least one chunk. The key-value pairs of the first group are stored in a first storage medium, and the key-value pairs of the second group are stored in a second storage medium. The number of accesses to the chunk corresponding to the key-value pairs in the first group is greater than the number of accesses to the chunk corresponding to the key-value pairs in the second group. The access speed of the first storage medium is faster than the access speed of the second storage medium.
5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the key-value pair corresponding to each chunk in the search results based on the first key-value cache includes: If the first key-value cache includes the key-value pair corresponding to the chunk in the search result, then the key-value pair corresponding to the chunk is retrieved from the first key-value cache; If the first key-value cache does not include the key-value pair corresponding to the chunk in the search results, then the key-value pair corresponding to the chunk is calculated.
6. The method according to claim 5, characterized in that, After calculating the key-value pairs corresponding to the chunk, the method further includes: If the chunk meets the preset conditions, the first key-value cache is updated based on the key-value pair corresponding to the chunk.
7. The method according to claim 5 or 6, characterized in that, The retrieval results include a first chunk and a second chunk. If the first key-value cache includes the key-value pair corresponding to the first chunk but does not include the key-value pair corresponding to the second chunk, then the operations of retrieving the key-value pair corresponding to the first chunk from the first key-value cache and calculating the key-value pair corresponding to the second chunk are performed in parallel.
8. A pre-filling device, characterized in that, The apparatus is used to perform the method according to any one of claims 1 to 7.
9. An electronic device, characterized in that, Used to perform the method according to any one of claims 1 to 7.
10. A computer program product, characterized in that, The computer program product includes instructions that, when executed by an electronic device, implement the method as described in any one of claims 1 to 7.