Retrieval-augmented generation
By retrieving and pre-filling recalled content from multiple dimensions through multiple parallel channels in chronological order, the problem of time-consuming generation and low hit rate in existing technologies is solved, and a more efficient generation process is achieved.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
- Filing Date
- 2026-01-16
- Publication Date
- 2026-07-23
AI Technical Summary
In existing retrieval enhancement generation methods, cluster partitioning is unrelated to retrieval queries, resulting in low hit rate of speculative generation and multiple invalid generation attempts. Furthermore, the top K update process depends on the order of retrieval results, which cannot effectively reduce generation time.
Multiple parallel channels are used to retrieve query requests from multiple dimensions, and the retrieved content is pre-filled according to the time order of the channel's returned content. The pre-filled results are stored using a caching device, and the retrieved content of the parallel channels is merged to optimize the generation process.
It reduces the waiting time during the retrieval enhancement request generation process, avoids redundant calculations in the pre-filling stage, and improves the prefix cache hit rate and generation efficiency.
Smart Images
Figure CN2026072997_23072026_PF_FP_ABST
Abstract
Description
Search Enhancement Generation Technical Field
[0001] This specification relates to one or more embodiments in the field of computer technology, and more particularly to search enhancement generation methods, apparatus, devices, and storage media. Background Technology
[0002] In some domains of Retrieval-Augmented Generation (RAG) applications, such as in finance, there is often a significant reliance on highly specialized content, which typically involves large datasets. Furthermore, to achieve good generation results, the parameters of the generation model are often quite large. In such cases, the effectiveness of low-level model optimization is limited. Existing techniques typically utilize a Dynamic Speculative Pipeline to improve the overall throughput of RAG and reduce generation time. This method involves cluster-by-cluster retrieval and immediate generation, using the current top K (best-to-last K) results for initial generation, and then updating the top K with the results of new cluster retrievals.
[0003] However, in this method, since the cluster partitioning and retrieval query are unrelated, the hit rate of speculative generation cannot be guaranteed, which can easily lead to multiple invalid generation and thus preempt normal generation requests. Furthermore, during the update of the top K, there is a strong dependence on the order of retrieval results, and in some cases, the prefix cache cannot be hit in the RAG cache, which cannot effectively reduce the generation time. Summary of the Invention
[0004] In view of this, one or more embodiments of this specification provide a retrieval enhancement generation method, apparatus, device, and storage medium to effectively reduce generation time.
[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions.
[0006] According to a first aspect of one or more embodiments of this specification, a retrieval enhancement generation method is proposed, comprising: retrieving a query request from multiple dimensions using multiple parallel channels; pre-filling the recall content of each channel according to the time order in which each channel returns the recall content, wherein the pre-filling result of each channel is stored in a cache device, and the pre-filling result of the channel is obtained based on the pre-filling result cached by the channel that returned the recall content previously; and obtaining the output result of a large language model based on the pre-filling result of the channel that returned the recall content last.
[0007] In some embodiments, the method further includes: determining the scenario to which the query request belongs; and determining the dimensions of each channel for retrieval based on the scenario.
[0008] In some embodiments, the step of pre-filling the recall content of each channel according to the time order of the recall content returned by each channel includes: when the time difference between the recall content returned by multiple channels is within a set range, merging the recall content of the multiple channels and then pre-filling based on the merged content.
[0009] In some embodiments, the step of pre-filling the recall content of each channel according to the time sequence of the recall content returned by each channel includes: when the length returned by a channel is less than a first set threshold, the recall content of the channels returned after that channel is merged until the length of the merged content is greater than the first set threshold, and then the merged content is pre-filled.
[0010] In some embodiments, the method further includes: storing the recalled content of the channel as a key and the corresponding pre-filled result as a value in a cache device; merging the recalled content of the channel with the recalled content of the previous channel, and using the merged content for pre-filling, wherein when a key is found in the merged content, the value corresponding to the key is obtained from the cache device as part of the pre-filled result.
[0011] In some embodiments, the method further includes: establishing a corresponding pre-filled queue for each query request; inputting the recall content of each channel obtained based on the query request into the pre-filled queue; wherein, after the recall content of one channel is pre-filled, the pre-filled queue pops out the recall content of the next channel.
[0012] In some embodiments, the method further includes: when multiple requests are received simultaneously, obtaining the following information for each request: the length L of the recalled content to be pre-filled in the current queue; the time P for pre-filling per unit length; the time Tx for completing all recalls of the request; the recall time T of the recalled content in the current queue; for each request, determining the maximum time saved Ts according to the formula Min(L*P,Tx-T); and determining the priority of the request according to the Ts corresponding to each request.
[0013] In some embodiments, the method further includes not pre-filling the request recall content if the following conditions are met: the sum of the lengths of the recall content for all channels is less than a second set threshold; or, the total duration of the completed recall for all channels is greater than a third set threshold.
[0014] In some embodiments, the method further includes: using a set number of pre-fill schedulers to transmit recalled content popped from a pre-fill queue to a module for pre-filling processing, wherein the set number is determined based on the concurrency supported by the large model.
[0015] According to a second aspect of one or more embodiments of this specification, a retrieval enhancement generation apparatus is proposed, comprising: a retrieval unit for retrieving query requests from multiple dimensions using multiple parallel channels; a pre-filling unit for pre-filling the recall content of each channel according to the time order in which the recall content is returned by each channel, wherein the pre-filling result of each channel is stored in a cache device, and the pre-filling result of the channel is obtained based on the pre-filling result cached by the channel that returned the recall content previously; and an output unit for obtaining the output result of a large language model based on the pre-filling result of the channel that returned the recall content last.
[0016] In some embodiments, the apparatus further includes a determining unit, configured to: determine the scenario to which the query request belongs; and determine the dimensions of each channel for retrieval based on the scenario.
[0017] In some embodiments, the pre-filling unit is specifically used to: merge the recall content of the multiple channels and pre-fill it according to the merged content when the time difference between the return of recall content from multiple channels is within a set range.
[0018] In some embodiments, the pre-filling unit is specifically used to: when the length returned by a channel is less than a first set threshold, merge the recall content of channels returned after that channel until the length of the merged content is greater than the first set threshold, and then pre-fill the merged content.
[0019] In some embodiments, the apparatus further includes a caching unit for: storing the recall content of the channel as a key and the corresponding pre-filled result as a value in the caching device; merging the recall content of the channel with the recall content of the previous channel, and using the merged content for pre-filling, wherein when a key is found in the merged content, the value corresponding to the key is obtained from the caching device as part of the pre-filled result.
[0020] In some embodiments, the apparatus further includes a queue unit for: establishing a corresponding pre-filled queue for each query request; inputting the recall content of each channel obtained based on the query request into the pre-filled queue; wherein, after the recall content of one channel is pre-filled, the pre-filled queue pops out the recall content of the next channel.
[0021] In some embodiments, the apparatus further includes a priority unit, configured to: when multiple requests are received simultaneously, obtain the following information for each request: the length L of the recalled content to be pre-filled in the current queue; the time P for pre-filling per unit length; the time Tx for completing all recalls of the request; the recall time T of the recalled content in the current queue; for each request, determine the maximum time saving Ts according to the formula Min(L*P,Tx-T); and determine the priority of the request according to the Ts corresponding to each request.
[0022] In some embodiments, the apparatus further includes a filtering unit for setting request recall content that meets the following conditions to not be pre-filled: the sum of the lengths of the recall content of all channels is less than a second preset threshold; or, the total duration of the completed recall of all channels is greater than a third preset threshold.
[0023] In some embodiments, the apparatus further includes a scheduling unit for: transmitting recall content popped from a pre-fill queue to a module for pre-filling processing using a set number of pre-fill schedulers, wherein the set number is determined based on the concurrency supported by the large model.
[0024] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method proposed in the above embodiments by executing the executable instructions.
[0025] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method proposed in the above embodiments.
[0026] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method proposed in the above embodiments.
[0027] The retrieval enhancement generation method proposed in this specification utilizes multiple parallel channels to retrieve query requests from multiple dimensions. Then, according to the chronological order of the recalled content returned by each channel, the recalled content of each channel is pre-filled. The pre-filled result of each channel is stored in a cache device, and the pre-filled result of that channel is obtained based on the pre-filled result cached by the channel that returned the recalled content previously. Finally, the output result of the large language model is obtained based on the pre-filled result of the channel that returned the last recalled content. Because pre-filling is performed according to the chronological order of the recalled content returned by the channels, the waiting time in a single retrieval enhancement generation request process is reduced, and the pre-filled result of the current channel is calculated based on the pre-filled result cached by the channel that returned the recalled content previously, avoiding redundant calculations in the pre-filling stage. Attached Figure Description
[0028] Figure 1 is an application scenario diagram of a retrieval enhancement generation method provided by an exemplary embodiment.
[0029] Figure 2 is a flowchart of a retrieval enhancement generation method provided in an exemplary embodiment.
[0030] Figure 3 is a schematic diagram of a pre-filling method provided in an exemplary embodiment.
[0031] Figure 4 is a schematic diagram of another pre-filling method provided in an exemplary embodiment.
[0032] Figure 5 is a block diagram of a retrieval enhancement generation apparatus provided in an exemplary embodiment.
[0033] Figure 6 is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation
[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0035] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.
[0036] To aid in understanding this manual, the technical terms related to this manual will be introduced first.
[0037] Retrieval-Augmented Generation (RAG) is a model that combines information retrieval and generation techniques. This technology generates answers or content by referencing information from external knowledge bases and can be used in various natural language processing tasks.
[0038] Prefilling: A technique used in the training and inference of large models. Before running a large model, some known or pre-computable information is input into the model to reduce the amount of real-time computation, reduce redundant calculations, and accelerate the inference process.
[0039] Decoding: In natural language processing tasks, the process by which a model generates an output sequence based on processed information (such as user input prompts or context). Decoding is typically performed after the prefilling stage.
[0040] In some RAG applications, such as in the financial sector, there is a heavy reliance on highly specialized content, which typically involves large datasets. Furthermore, to achieve good generation results, the parameters of the generation model are often quite large. In such cases, the effectiveness of optimizing the underlying model is limited.
[0041] Existing technologies typically utilize dynamic speculative pipelines to improve overall RAG throughput and reduce generation time. This method is generally implemented according to the following steps.
[0042] Step 1: Divide the vector search into several clusters, and traverse each cluster to perform the search;
[0043] Step 2: After obtaining the top K returned by the first cluster, send them to the model in advance for generation; where top K refers to the K best recalled items.
[0044] Step 3: After completing the retrieval of the second cluster, update the top K. If the top K changes, the generation in Step 2 is invalid and needs to be regenerated.
[0045] Step 4: Repeat steps 2 and 3 until the retrieval of the last cluster is completed.
[0046] The above method has the following two problems.
[0047] Question 1: Since cluster partitioning and retrieval queries are unrelated, the hit rate of speculative generation cannot be guaranteed, which can easily lead to multiple invalid generation attempts, thereby preempting normal generation requests.
[0048] Question 2: During the top K update process, there is a strong dependence on the order of the search results. If a single result is hit but the order is different, for example, the first cluster returns D1 and D2, and the second cluster returns D2 and D3, then the prefix cache cannot be hit in the RAG cache, and the generation time cannot be effectively reduced.
[0049] To address this problem, this application proposes a search enhancement generation scheme.
[0050] Figure 1 illustrates an application scenario of a retrieval enhancement generation method according to an embodiment of this application. Figure 1 includes a service terminal 10 and an application program 110, a large language model 120, and a caching device 130 installed in the service terminal.
[0051] The service terminal 10 is a device or software entity that integrates service provision and request processing functions. It can run in both client and server environments. It can receive and process requests from users, execute corresponding service logic, and return the processing results to the requester.
[0052] Application 110 is a software program that enables end users to interact with a pre-trained large language model 120 by providing a user-friendly interface.
[0053] Big Language Model 120 is a highly complex artificial intelligence system that learns language knowledge from large amounts of text using deep learning technology and can be applied to a variety of natural language processing tasks, such as virtual assistants and chatbots, customer service, education, and data analysis.
[0054] The caching device 130 can be used to cache the output, intermediate results, or parameters of the large language model 120 to improve the efficiency and response speed of the model service.
[0055] It should be noted that adding additional modules to or removing individual modules from the illustrated environment does not change the underlying concept of the example embodiments of this application.
[0056] It should be noted that the retrieval enhancement generation method proposed in this application is not only applicable to the application scenario shown in Figure 1, but also applicable to any device that has retrieval enhancement generation requirements.
[0057] Figure 2 shows a flowchart of a retrieval enhancement generation method provided in an embodiment of this application, which is executed by the service terminal 10 in Figure 1. As shown in Figure 2, the method includes steps 201 to 203.
[0058] Step 201: Use multiple parallel channels to retrieve the query request from multiple dimensions.
[0059] Parallel channels refer to independent processing paths that can run concurrently, with each channel processing data independently. These channels can be physical, such as multiple processors or servers, or logical, such as multiple threads or processes on the same processor.
[0060] By using multiple parallel channels to retrieve query requests from multiple dimensions, the query requests can be analyzed from different angles or features simultaneously, which allows for a more comprehensive understanding of the user's intent.
[0061] In some embodiments, retrieval can be performed from dimensions such as text content, semantic understanding, user behavior, geographic location, time information, etc. Each channel independently retrieves relevant information from the corresponding dimension for the query assigned to it.
[0062] The specific dimensions to be searched can be determined based on the context of the query request. For each context, a mapping relationship between each channel and the search dimensions can be pre-established. Therefore, upon receiving a query request, the context of the query request can be determined first, and then the dimensions of each channel to be searched can be determined based on the context.
[0063] Taking the financial sector as an example, the scenario to which a query request pertains may involve Q&A between financial users and financial product customer service, investment advice, financial news analysis, and financial product recommendations. For instance, assuming the scenario to which the query request pertains is determined to be financial product recommendations, then based on the pre-established mapping relationship between various channels and search dimensions, it can be determined that channel 1 searches from the perspective of financial viewpoints, channel 2 searches from the perspective of financial information, channel 3 searches from the perspective of market dynamics, and so on.
[0064] Step 202: Pre-fill the recall content for each channel according to the time sequence of the recall content returned by each channel.
[0065] The pre-filled result for each channel is stored in a cache device (also known as a prefix cache), and the pre-filled result for that channel is obtained based on the pre-filled result cached by the channel that returned the recall content.
[0066] In this embodiment, each channel retrieves the query content from different dimensions. Since the data volume of the data source or the amount of recalled content retrieved by each channel may differ, or the retrieval algorithms used by each channel may differ, or the resource allocation of each channel may differ, the time it takes for each channel to return the recalled content may vary significantly. For the channel that returns the recalled content first, the content is added to the prompt for pre-filling calculation, and the pre-filling result of that channel is stored in the cache. For the next channel that returns the recalled content, after the previous channel completes its pre-filling, the recalled content of the previous channel is merged with the recalled content of this channel for joint pre-filling calculation. Since the pre-filling result of the first part of the merged recalled content has already been calculated, this channel only performs pre-filling calculation on its own recalled content, retrieves the pre-filling result of the previous channel from the cache, and combines it with the pre-filling result corresponding to its own recalled content to obtain the final pre-filling result output by that channel. Subsequent channels returning recalled content all obtain their pre-filling results in this manner.
[0067] Step 203: Based on the pre-filling results of the channel that returned the final recalled content, obtain the output results of the large language model.
[0068] After obtaining the pre-filled result of the channel that returned the final recalled content, the pre-filled result is input into the large language model. The large language model decodes the pre-filled result and converts the implicit state inside the model into the final text output result.
[0069] Refer to Figure 3, which illustrates the pre-filling method provided in this embodiment of the present disclosure. Assume that a query is retrieved from three different dimensions using parallel channels 1, 2, and 3, with retrieved contents R1, R2, and R3 respectively, and the order in which the retrieved contents are returned is R1, R3, and R2. After channel 1 returns the retrieved content R1, pre-filling calculation is immediately performed on R1 to obtain the pre-filled result P1. Next, channel 3 returns the retrieved content R3. After the pre-filling calculation of channel 1 is completed, R1 and R3 are merged, and pre-filling calculation is performed on the merged result. Since the pre-filled result of R1 is already stored in the cache device, the cached result can be directly called, and only the pre-filled result P3 of R3 needs to be calculated. After completing the calculation of R3, the pre-filled result (P1+P3) corresponding to channel 3 is obtained. Then, channel 2 returns the retrieved content R2. After the pre-filling calculation of channel 3 is completed, R1, R3, and R2 are merged, and pre-filling calculation is performed on the merged result. During the calculation, the pre-filled results (P1+P3) corresponding to R1 and R3 cached in channel 3 are directly called, and then combined with the pre-filled result P2 corresponding to R2 to obtain the pre-filled result (P1+P3+P2) corresponding to channel 3. After obtaining the pre-filled result of channel 2, the pre-filled result is decoded to obtain the final output of the large model.
[0070] The retrieval enhancement generation method proposed in this specification pre-fills the data according to the chronological order of the recalled content returned by each channel. This allows for pre-filling calculations to be performed as soon as a channel returns recalled content, eliminating the need to wait for all channels to complete their recall before pre-filling. By merging the recall and generation stages, this method can fully utilize the time difference between multiple recalls, reducing the waiting time in a single RAG request, especially when there are significant differences in recall time. Furthermore, because pre-filling is performed according to the chronological order of the recalled content returned by each channel, it ensures that the pre-filling result of the current channel can be obtained using the pre-filling result cached by the channel that returned recalled content previously, avoiding redundant calculations in the pre-filling stage.
[0071] In some embodiments, when the time difference between the return of recalled content from multiple channels is within a set range, the recalled content from the multiple channels is merged and then pre-filled based on the merged content.
[0072] If the recall times of multiple channels are very close, and there is no need to wait too long for the recall, the contents of these channels can be merged first, and then pre-filled together based on the merged contents to improve the efficiency of pre-filling calculation.
[0073] In some embodiments, when the length returned by a channel is less than a set threshold, the recalled content of channels returned after that channel is merged until the length of the merged content exceeds a first set threshold, at which point the merged content is pre-padded. For example, the first set threshold can be set to 500 characters, and the merged content is pre-padded only when the length of the merged content exceeds 500 characters.
[0074] If the recalled content for a channel is very limited, the overhead of pre-filling calculations for that channel alone may outweigh the time saved. To address this issue, if the length of the recalled content for a channel is determined to be less than a first preset threshold, pre-filling calculations can be temporarily withheld, and the system can wait for the recalled content from the next channel. After merging the recalled content from these two channels, it is compared with the first preset threshold. If it is still less than the first preset threshold, the system continues to merge the recalled content from the next channel until the length of the merged content exceeds the first preset threshold. The merged content is then used for pre-filling, effectively reducing the system overhead during the search enhancement generation process.
[0075] Furthermore, if the sum of the lengths of the recalled content across all channels is less than a second set threshold, or if the total recall time across all channels exceeds a third set threshold, then the recalled content is not pre-filled and is directly encoded to obtain the output of the large language model. For example, the second set threshold can be set to 2000 characters, and the third set threshold can be set to 2 seconds. That is, by setting the above two thresholds, requests with too short recalled content or too long recall time are eliminated to avoid the overhead of pre-filling calculations exceeding the overhead of direct encoding, thus negatively impacting the system.
[0076] In some embodiments, the recalled content of the channel is stored in a cache device as the key and the corresponding pre-filled result is stored as the value. When the recalled content of the channel is merged with the recalled content of the previous channel and the merged content is used for pre-filling, when a key is found in the merged content, the value corresponding to the key is obtained from the cache device as part of the pre-filled result.
[0077] In this embodiment, the pre-filling is performed according to the time sequence of the recall content returned by each channel, and the recall content is used as the key and the corresponding pre-filling result is used as the value for caching. Therefore, when each channel merges the recall content of the previous channel for pre-filling calculation, it can ensure that the pre-filling result corresponding to the recall content of the previous channel can be retrieved from the cache device, avoiding repeated calculations in the pre-filling stage and effectively reducing system overhead.
[0078] Since generative models typically require significant computational resources, invoking them incurs additional overhead. To manage generative model invocations, a scheduling queue can be introduced in this embodiment to determine when to initiate segmented pre-filling.
[0079] Specifically, when entering the pre-filled queue, the queue can be designed according to the query requests. That is, a corresponding pre-filled queue is established for each query request, and the recall content of each channel obtained based on the query request is input into the same pre-filled queue. Furthermore, after the recall content of one channel is pre-filled, the recall content of the next channel is popped out of the pre-filled queue.
[0080] Referring to Figure 4, which illustrates the pre-filling method, each query request corresponds to a pre-filling queue. Content retrieved from multiple channels for a single query request enters the same pre-filling queue. For the retrieved content in each pre-filling queue, a pre-filling scheduler (prefiling worker) transmits the content popped from the queue to the module used for pre-filling processing.
[0081] As shown in Figure 4, for query request 1, the recall content R11, R12, and R13 returned from the three channels respectively enter pre-fill queue 1; for query request 2, the recall content R21, R22, and R23 returned from the three channels respectively enter pre-fill queue 2; and for query request 3, the recall content R31, R32, and R33 returned from the three channels respectively enter pre-fill queue 3. For each pre-fill queue, after the recall content of one channel is pre-filled, the recall content of the next channel is popped. For example, suppose that for query request 1, the recall content R11 of channel 1 has been returned, while other channels are still waiting, then R11 will be sent to the pre-fill scheduler 1 for pre-fill calculation in the next scheduling. Suppose that for query request 2, the recall content R21 of channel 1 has been pre-filled, the recall content R22 of channel 2 has been returned, while the third channel is still waiting, then R21 will be sent to the pre-fill scheduler 2 for pre-fill calculation in the next scheduling. Suppose that for query request 3, the recall content R21 of channel 1 has been pre-filled, the recall content R22 of channel 2 and the recall content R23 of channel 3 have been returned, and the time difference between their returns is within a set range, for example, the interval is less than 0.1 seconds, then R32 and R33 will be sent to the pre-fill scheduler 3 together for pre-fill calculation in the next scheduling, instead of being scheduled separately.
[0082] Since the recalled content returned by each channel for a request is sent into the pre-scheduling queue in the order of return, and the recalled content returned first is popped out of the queue for pre-filling calculation, the priority device in this embodiment does not change the priority of the recalled content in the queue, and is processed according to the first-in-first-out principle in actual execution.
[0083] Since each pre-filled queue pops up the next channel's recall content only after the recall content of one channel has been pre-filled, it can ensure that only one pre-filled content is being calculated for a query request at a time, avoiding unnecessary parallel computation, making more efficient use of computing resources, and dynamically assembling the prompt content according to the first-in-first-out order, thus increasing the hit rate of the prefix cache to 100%.
[0084] In some embodiments, the number of pre-filled schedulers can be determined based on the concurrency supported by the large model.
[0085] Assuming the large model supports a concurrency of n, then n pre-fill schedulers can be set up to extract content from the pre-fill queue, ensuring that at most n requests enter the pre-fill settlement at the same time. This guarantees that there is sufficient capacity to handle such pre-fill requests during the model generation phase.
[0086] When multiple requests are received simultaneously, to ensure the cost-effectiveness of pre-filling computation, decisions need to be made regarding the recall time and recall length for each request. This avoids requests with extremely small time differences and very limited recall content in multi-path recall, as the additional overhead of these requests may outweigh the time saved by pre-filling. It's important to note that "simultaneously" here refers to the receiving time difference being within a set range.
[0087] In some embodiments, when multiple requests are received simultaneously, the following information can be obtained for each request: the length L of the recalled content to be pre-filled in the current queue; the time P for pre-filling per unit length; the time Tx for completing all recalls of the request; the recall time T of the recalled content in the current queue; for each request, the maximum time saving Ts is determined according to the formula Min(L*P,Tx-T); the priority of each request is determined according to the maximum time saving.
[0088] Among them, L*P is the time taken to pre-fill the recalled content in the current queue, and Tx - T is the maximum time that can be saved by enabling pre-filling, that is, the waiting time that can be avoided if pre-filling is performed immediately. In the actual execution process, since Tx is unpredictable, it can be estimated based on the average time consumption of each recall channel and according to the recall channel matched by the current request. When L*P < Tx - T, among multiple requests, if the value of L*P of a certain request is larger, it means that this request has greater potential time savings. Therefore, the larger the value of L*P, the higher the priority of this request.
[0089] When L*P ≥ Tx - T, the time that can be saved by performing the pre-filling operation in advance is (Tx - T), which means that this request has greater potential time savings. Therefore, the larger the value of Tx - T, the higher the priority of this request.
[0090] By determining the priority of requests through the above method, when multiple requests are received simultaneously, requests with higher cost performance can be selected for calculation, improving the overall response speed of the system and optimizing resource utilization.
[0091] Figure 5 is a block diagram of a retrieval enhanced generation device provided by an exemplary embodiment. As shown in Figure 5, the device includes a retrieval unit 501, a pre-filling unit 502, and an output unit 503.
[0092] The retrieval unit 501 is used to retrieve the query request from multiple dimensions respectively using multiple parallel channels.
[0093] The pre-filling unit 502 is used to pre-fill the recalled content of each channel in the time order of the recalled content returned by each channel. Among them, the pre-filling result of each channel is stored in the cache device, and the pre-filling result of this channel is obtained based on the pre-filling result cached by the previous channel that returned the recalled content.
[0094] The output unit 503 is used to obtain the output result of the large language model based on the pre-filling result of the channel that finally returns the recalled content.
[0095] In some embodiments, the device further includes a determination unit, which is used to: determine the scenario to which the query request belongs; determine the dimensions of each channel for retrieval according to the scenario.
[0096] In some embodiments, the pre-filling unit is specifically used to: when the time difference between the recalled content returned by multiple channels is within a set range, merge the recalled content of the multiple channels and then perform pre-filling according to the merged content.
[0097] In some embodiments, the pre-filling unit is specifically used to: when the length returned by a channel is less than a first set threshold, merge the recall content of channels returned after that channel until the length of the merged content is greater than the first set threshold, and then pre-fill the merged content.
[0098] In some embodiments, the apparatus further includes a caching unit for: storing the recall content of the channel as a key and the corresponding pre-filled result as a value in the caching device; merging the recall content of the channel with the recall content of the previous channel, and using the merged content for pre-filling, wherein when a key is found in the merged content, the value corresponding to the key is obtained from the caching device as part of the pre-filled result.
[0099] In some embodiments, the apparatus further includes a queue unit for: establishing a corresponding pre-filled queue for each query request; inputting the recall content of each channel obtained based on the query request into the pre-filled queue; wherein, after the recall content of one channel is pre-filled, the pre-filled queue pops out the recall content of the next channel.
[0100] In some embodiments, the apparatus further includes a priority unit, configured to: when multiple requests are received simultaneously, obtain the following information for each request: the length L of the recalled content to be pre-filled in the current queue; the time P for pre-filling per unit length; the time Tx for completing all recalls of the request; the recall time T of the recalled content in the current queue; for each request, determine the maximum time saving Ts according to the formula Min(L*P,Tx-T); and determine the priority of the request according to the Ts corresponding to each request.
[0101] In some embodiments, the apparatus further includes a filtering unit for setting request recall content that meets the following conditions to not be pre-filled: the sum of the lengths of the recall content of all channels is less than a second preset threshold; or, the total duration of the completed recall of all channels is greater than a third preset threshold.
[0102] In some embodiments, the apparatus further includes a scheduling unit for: transmitting recall content popped from a pre-fill queue to a module for pre-filling processing using a set number of pre-fill schedulers, wherein the set number is determined based on the concurrency supported by the large model.
[0103] Figure 6 is a schematic structural diagram of a device provided in an exemplary embodiment. Referring to Figure 6, at the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and may also include other hardware required for services. One or more embodiments of this specification can be implemented in software, for example, the processor 602 reads the corresponding computer program from the non-volatile memory 610 into the memory 608 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0104] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.
[0105] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0106] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0107] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0108] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0109] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0110] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0111] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."
[0112] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.
Claims
1. A retrieval enhancement generation method, comprising: The query request is retrieved from multiple dimensions using multiple parallel channels. According to the time order of the recall content returned by each channel, the recall content of each channel is pre-filled. The pre-filling result of each channel is stored in the cache device, and the pre-filling result of the channel is obtained based on the pre-filling result cached by the previous channel that returned the recall content. The output of the large language model is obtained based on the pre-filled results of the channel that returned the last recalled content.
2. The method according to claim 1, further comprising: Determine the scenario to which the query request belongs; The dimensions of each channel for retrieval are determined based on the scenario.
3. The method according to claim 1, wherein pre-filling the recall content of each channel according to the time sequence of the recall content returned by each channel includes: If the time difference between the return of recall content from multiple channels is within a set range, the recall content from the multiple channels is merged and then pre-filled based on the merged content.
4. The method according to claim 1, wherein pre-filling the recall content of each channel according to the time sequence of the recall content returned by each channel includes: When the length returned by a channel is less than a first set threshold, the recall content of the channels returned after that channel will be merged until the length of the merged content is greater than the first set threshold, at which point the merged content will be pre-filled.
5. The method according to claim 1, further comprising: The recalled content of the channel is used as the key and the corresponding pre-filled result is used as the value to store the data in the cache device; The recall content of the channel is merged with the recall content of the previous channel, and the merged content is used for pre-filling. When a key is found in the merged content, the value corresponding to the key is obtained from the cache device as part of the pre-filling result.
6. The method according to claim 1, further comprising: Establish a corresponding pre-populated queue for each query request; The retrieved content for each channel obtained based on the query request is input into the pre-filled queue; The pre-filled queue pops up the recall content for the next channel after the recall content for one channel has been pre-filled.
7. The method according to claim 1, further comprising: When multiple requests are received simultaneously, the following information is obtained for each request: The length L of the recall content to be pre-filled in the current queue; The time P for pre-filling per unit length; The time Tx for completing all recalls requested; The recall time T for the content currently being recalled in the queue; For each request, the maximum time saving Ts is determined according to the formula Min(L*P,Tx-T); The priority of each request is determined based on its corresponding Ts.
8. The method of claim 1, further comprising not pre-populating request recall content that meets the following conditions: The sum of the lengths of the recalled content from all channels is less than a second set threshold; or, The total time for completing the recall across all channels exceeds the third set threshold.
9. The method according to claim 1, further comprising: A set number of pre-filled schedulers are used to transmit the recalled content popped from the pre-filled queue to the module for pre-filling processing, wherein the set number is determined according to the concurrency supported by the large language model.
10. A retrieval enhancement generation apparatus, comprising: The retrieval unit is used to retrieve query requests from multiple dimensions using multiple parallel channels; The pre-filling unit is used to pre-fill the recall content of each channel according to the time order of the recall content returned by each channel. The pre-filling result of each channel is stored in the cache device, and the pre-filling result of the channel is obtained based on the pre-filling result cached by the previous channel that returned the recall content. The output unit is used to obtain the output of the large language model based on the pre-filled results of the channel that returned the last recalled content.
11. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-9 by executing the executable instructions.
12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.
13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.