Data processing method and system

Abstract

Embodiments of the present description provide a data processing method and system, which are applied to the field of computers. The method comprises: determining a current query, and rewriting the current query on the basis of historical query data corresponding to the current query, and obtaining a target query; on the basis of the target query, performing retrieval from a first data source, obtaining a first retrieval result, and determining a target search policy on the basis of an evaluation result obtained through evaluation of the first retrieval result; on the basis of the target search policy and the target query, performing retrieval from a second data source, obtaining a second retrieval result, and generating a prompt text on the basis of the first retrieval result and the second retrieval result; and on the basis of the target query and the prompt text, obtaining a target query result corresponding to the current query. The target search policy is determined on the basis of the evaluation result, so that accurate retrieval can be achieved, and the retrieval results are obtained from a plurality of data sources to generate the prompt text, so that an accurate target query result is obtained.

Description

Data processing methods and systems

[0001] This disclosure claims priority to Chinese Patent Application No. 202411586428.9, filed with the China Patent Office on November 7, 2024, entitled “Data Processing Method and System”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing method and system. Background Technology

[0003] With the rapid development of information technology, the amount of data has exploded, making it a significant challenge to efficiently and accurately extract the necessary knowledge from massive amounts of information. Traditional information retrieval technologies have certain limitations when processing large-scale, complex datasets, especially in understanding user needs and providing personalized services.

[0004] In recent years, large models have made significant progress in the field of natural language processing due to their powerful language understanding and generation capabilities. However, large models have limitations in retrieving and generating answers and cannot be personalized for users. Therefore, the relevance and accuracy of the answers provided need to be improved. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical shortcomings of existing technologies, such as the limitations of large models in generating answers and the inability to personalize them for users, resulting in insufficient relevance and accuracy of the provided answers.

[0006] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0007] Determine the current query, and rewrite the current query based on the historical query data corresponding to the current query to obtain the target query;

[0008] Based on the target query, a first search result is obtained by retrieving from the first data source, and a target search strategy is determined based on the evaluation result obtained by evaluating the first search result.

[0009] Based on the target search strategy and the target query, a second search result is obtained from the second data source, and a prompt text is generated based on the first search result and the second search result;

[0010] Based on the target query and the prompt text, obtain the target query result corresponding to the current query.

[0011] According to a second aspect of the embodiments of this specification, a data processing system is provided, including a rewriting unit, an evaluation unit, a generation unit, and an acquisition unit, wherein...

[0012] The rewriting unit is used to determine the current query and rewrite the current query based on the historical query data corresponding to the current query to obtain the target query;

[0013] The evaluation unit is used to retrieve a first search result from a first data source based on the target query, and to determine a target search strategy based on the evaluation result obtained by evaluating the first search result.

[0014] The generation unit is configured to retrieve a second search result from a second data source based on the target search strategy and the target query, and generate a prompt text based on the first search result and the second search result;

[0015] The obtaining unit is used to obtain the target query result corresponding to the current query based on the target query and the prompt text.

[0016] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0017] Memory and processor;

[0018] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0019] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0020] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0021] This specification provides a data processing method in one embodiment. For a user's current query, the method can rewrite the current query based on historical query data to obtain a target query that more closely reflects the user's true intent. For the target query, a first search result can be retrieved from a first data source. This first search result can be evaluated to obtain an evaluation result. If a target search strategy can be determined based on the evaluation result, a more accurate search can be achieved, obtaining a first search result more relevant to the target query. Furthermore, a second search result can be obtained from a second data source. By obtaining search results from multiple data sources, and generating prompt text based on the first and second search results from the first and second data sources, a more accurate target query result can be obtained through the target query and the related prompt text. Attached Figure Description

[0022] Figure 1 is a schematic diagram of a data processing method provided in one embodiment of this specification;

[0023] Figure 2 is a flowchart of a data processing method provided in one embodiment of this specification;

[0024] Figure 3 is a schematic diagram of the processing procedure of a data processing method provided in one embodiment of this specification;

[0025] Figure 4 is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification;

[0026] Figure 5 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0027] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0028] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0030] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0032] Agent: Intelligent agent.

[0033] RAG: Retrieval Augmented Generation.

[0034] This specification provides a data processing method, and also relates to a data processing system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0035] Referring to Figure 1, Figure 1 shows a schematic diagram of a data processing method provided according to an embodiment of this specification.

[0036] The specific data processing method is applied to the data processing system. The following is a detailed description of the data processing method applied to the data processing system.

[0037] In practical applications, the data processing method is implemented using the terminal device 102 and the server 104. The terminal device 102 is used to send the current query to the server 104, such as the current query being "how many months are there in a year". In practical applications, users can input the current query into the terminal device 102 via text or voice. If voice is used, the terminal device 102 will also include a corresponding voice processing component, such as a voice parsing, voice-to-text, and voice synthesis module, to convert the user's voice input into text. This specification does not impose any restrictions on this.

[0038] A data processing system is deployed on server 104. When server 104 receives a current query sent by end device 102, the data processing system will apply data processing methods and perform the following processing:

[0039] The current query is determined, and the current query is rewritten based on the historical query data corresponding to the current query to obtain the target query. According to the target query, a first search result is obtained by searching from a first data source, and a target search strategy is determined based on the evaluation result obtained by evaluating the first search result. According to the target search strategy and the target query, a second search result is obtained by searching from a second data source, and a prompt text is generated based on the first search result and the second search result. According to the target query and the prompt text, the target query result corresponding to the current query is obtained. If the target query result is "There are 12 months in a year", the target query result is returned to the end device 102.

[0040] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0041] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0042] This specification provides a data processing method in one embodiment. For a user's current query, the method can rewrite the current query based on historical query data to obtain a target query that more closely reflects the user's true intent. For the target query, a first search result can be retrieved from a first data source. This first search result can be evaluated to obtain an evaluation result. If a target search strategy can be determined based on the evaluation result, a more accurate search can be achieved using different search strategies, obtaining a first search result more relevant to the target query. Furthermore, a second search result can be obtained from a second data source. By obtaining search results from multiple data sources, and generating prompt text based on the first and second search results from the first and second data sources, a more accurate target query result can be obtained through the target query and the related prompt text.

[0043] Referring to Figure 2, which shows a flowchart of a data processing method provided in one embodiment of this specification, the method specifically includes the following steps.

[0044] Step 202: Determine the current query, and rewrite the current query based on the historical query data corresponding to the current query to obtain the target query.

[0045] The current query can be understood as the user query or question currently received, representing the information the user currently wants to find; historical query data can be understood as all query records made by the user in the past period before the current query, including the user's query question and the output query results, and may also include the user's feedback information on the output query results, without limitation; this historical query data can be used to analyze the user's query habits, interests and preferences, and query intent, etc.

[0046] A target query can be understood as a rewritten user query or question that is closer to the user's actual intent, which helps improve the accuracy and relevance of search results.

[0047] Specifically, upon receiving the current query, it can be rewritten based on corresponding historical query data. For example, pronouns in the current query can be replaced according to historical query data to obtain a target query that is closer to the user's actual intent. Alternatively, grammatical errors in the current query can be corrected using historical query data to make the current query have complete and clear semantics. Furthermore, when rewriting the current query using historical query data, the relevant content of the current query can be expanded using historical query data to obtain more relevant search results.

[0048] For example, if the historical query data is "What's the weather like in Beijing today? - It's cloudy in Beijing today, and it's recommended to wear a light jacket," and the user wants to further query the weather in Shanghai, based on the user's language habits, the user usually won't type "What's the weather like in Shanghai today?", but will continue from the previous question, such as "What about Shanghai?". Therefore, the current query is rewritten using the historical query data, changing the current query to the target query "What's the weather like in Shanghai today?".

[0049] The data processing method provided in the embodiments of this specification rewrites the current query by considering historical query data, thereby obtaining a target query that is closer to the user's actual intent. This allows for a better understanding of the user's personalized needs and enables subsequent search processes to obtain more relevant and accurate search results based on the target query, providing more personalized answers.

[0050] In one or more embodiments of this specification, through the interaction between the client and the server, users can obtain target query results more efficiently and accurately based on the current query, thereby improving the user's query experience and satisfaction. Specific implementation methods are described below:

[0051] Determining the current query includes:

[0052] Receive the current query sent by the client, wherein the current query is determined by the client based on interactive operations on the user interface;

[0053] Specifically, users can use the client's user interface to click or enter the current query. Based on the user's interaction on the user interface, the client sends the current query to the server, and the server receives the current query sent by the client.

[0054] The data processing method provided in the embodiments of this specification enables users to send current queries more conveniently through the client and the server through the interaction between the client and the server, thereby improving the user's query experience and satisfaction.

[0055] Step 204: Based on the target query, retrieve the first search result from the first data source, and determine the target search strategy based on the evaluation result obtained by evaluating the first search result.

[0056] The first data source can be understood as the data set or database used when performing an initial retrieval of the target query. It can be any repository containing information that may satisfy the target query. In this embodiment, the first data source is a local knowledge base. The first search result can be understood as the result retrieved from the first data source based on the target query.

[0057] The evaluation result can be understood as the evaluation result obtained by assessing the quality, relevance, completeness, etc. of the first search result. The evaluation result can be represented in the form of a score, and the evaluation result can be obtained by processing the first search result using a pre-trained model, or by human evaluation, without any restrictions.

[0058] A target search strategy can be understood as a strategy for further retrieval or querying based on the evaluation results. For example, a target search strategy could be to modify the query strategy (adjust the target query) or to continue the search strategy (use the next data source to search).

[0059] Specifically, when a target query is obtained, the target query can first be retrieved in the local knowledge base to obtain the first search results related to the target query. Furthermore, the first search results can be evaluated, and the target search strategy can be determined based on the evaluation results: modify the query strategy or continue the search strategy.

[0060] In one or more embodiments of this specification, a target intent is obtained through a target query, and a first search result is retrieved from a first data source based on the target intent. The specific implementation method is as follows:

[0061] The step of retrieving a first search result from a first data source based on the target query, and determining a target search strategy based on an evaluation result obtained by evaluating the first search result, includes:

[0062] Based on the target query, determine the target intent corresponding to the target query;

[0063] Based on the stated target intent, a first search result is obtained by retrieving from a first data source, and a target search strategy is determined based on the evaluation result obtained by evaluating the first search result.

[0064] In this context, target intent can be understood as the specific purpose or need that a user wants to achieve, determined by analyzing the target query. For example, if the target query is "best-selling smartphones in 2023", the target intent could be to obtain a list of smartphones ranked by sales volume.

[0065] Specifically, by analyzing the target query, we can determine the user's target intent, more accurately understand the information the user wants to obtain, and retrieve the first search results from the first data source based on the target intent. We can ensure that the retrieved content matches the user's target intent, evaluate the first search results in terms of accuracy, relevance, and completeness to determine whether they meet the user's needs, and determine the next target search strategy based on the evaluation results to further optimize the search process.

[0066] The data processing method provided in the embodiments of this specification can ensure that the first search result is more in line with the user's needs by determining the target intent and conducting targeted retrieval, and can improve the retrieval accuracy by evaluating the first search result and determining the target search strategy based on the obtained evaluation result, thus ensuring that the information obtained by retrieval is more accurate.

[0067] In one or more embodiments of this specification, the target search strategy includes modifying the query strategy or continuing the search strategy; by evaluating the relevance and quality of the first search result, an evaluation result corresponding to the first search is obtained, and then the corresponding target search strategy is determined based on the evaluation result. Specific implementation methods are as follows:

[0068] The step of determining the target search strategy based on the evaluation result obtained by evaluating the first search result includes:

[0069] The relevance and quality of the first search result are evaluated to obtain the evaluation result, and the modified query strategy or the continued search strategy is determined based on the evaluation result.

[0070] Specifically, relevance assessment can be understood as judging the degree of matching between the first search result and the user's query intent. For example, highly relevant results are closer to the user's actual needs. Quality assessment can be understood as a comprehensive evaluation of the accuracy, completeness, timeliness, and other aspects of the first search result. High-quality results are usually more accurate, comprehensive, and timely.

[0071] In practice, a specific evaluation model can be used to assess the relevance and quality of the first search result, so as to obtain the evaluation result efficiently and accurately, and then determine the specific strategy to modify the query or continue the search based on the evaluation result.

[0072] In practical applications, the first search result is evaluated for relevance to determine whether it matches the user's query intent. The first search result is also evaluated for quality to ensure its accuracy, completeness, and / or timeliness. The final evaluation result is obtained by combining the relevance and quality evaluations (e.g., using a weighted summation method).

[0073] Based on the evaluation results, determine the next target search strategy. If the evaluation results indicate that the first search results are not ideal (e.g., low relevance, poor quality), the target search strategy is to modify the query strategy to adjust the target query. If the evaluation results indicate that the first search results basically meet the requirements, a continued search strategy can be chosen, such as using different data sources to further search the target query.

[0074] The data processing method provided in the embodiments of this specification can dynamically determine or adjust the target search strategy based on the evaluation results by evaluating the relevance and quality of the search results and deciding whether to continue searching or modify the query, thereby improving the intelligence level of the system and enhancing its flexibility and adaptability.

[0075] Step 206: Based on the target search strategy and the target query, retrieve the second search result from the second data source, and generate a prompt text based on the first search result and the second search result.

[0076] In practical applications, a second data source is used to retrieve the target query to obtain the second search results. The first search results are then verified or supplemented by different data sources to improve the comprehensiveness and accuracy of the search. The first and second search results are combined to generate prompt text, so that when the target query and prompt text are input into the language model, more accurate target query results can be obtained.

[0077] In one or more embodiments of this specification, the target search strategy includes a modified query strategy or a continued search strategy. If the target search strategy is determined to be a modified query strategy based on the evaluation result, the current query can be rewritten to obtain a modified query. This modified query is then used to retrieve data from a first data source, obtain a first search result, and evaluate the first search result. By repeating the above steps, a first search result that achieves the expected effect is obtained, and the target search strategy is determined to be a continued search strategy based on the evaluation result. Specific implementation methods are as follows:

[0078] The step of obtaining a second search result from a second data source based on the target search strategy and the target query includes:

[0079] If the target search strategy is determined to be the modified query strategy, the current query is rewritten based on the historical query data corresponding to the current query to obtain the modified query.

[0080] The modified query is identified as the target query, and the step of retrieving the first search result from the first data source based on the target query is continued until the target search strategy is determined to be the continuing search strategy based on the evaluation result obtained by evaluating the first search result. If the target search strategy is determined to be the continuing search strategy, the second search result is obtained from the second data source based on the target query.

[0081] In practical applications, if the evaluation result does not meet expectations, the current query can be rewritten based on the historical query data corresponding to the current query to obtain a modified query. This modified query can be an optimized query that is different from the previous target query. The modified query is then used as the target query, and a first search result corresponding to the target query is obtained from the first data source. This first search result is evaluated to obtain an evaluation result. If the target search strategy is determined to be a continued search strategy based on the evaluation result, the above loop ends, and a second search result is obtained from a second data source that is different from the first data source based on the continued search strategy and the target query.

[0082] In other words, if the first search result does not meet expectations, the current query can be rewritten multiple times to generate a prompt text based on the first search result that meets expectations and the second search result obtained from the second data source.

[0083] The data processing method provided in the embodiments of this specification, by employing a multi-round rewriting approach, can dynamically adjust the current query based on historical query data, thereby generating a more accurate target query, better understanding the user's intent, gradually optimizing search results, improving the accuracy and relevance of the retrieval, and thus improving the retrieval effect.

[0084] In one or more embodiments of this specification, the target search strategy includes a continued search strategy; obtaining a second search result from a second data source different from the first data source; and generating prompt text based on the first search result obtained from the different data source and the second search result. Specific implementation methods are described below:

[0085] Based on the target search strategy and the target query, a second search result is obtained from the second data source, including:

[0086] Based on the continued search strategy and the target query, the second search result is obtained by retrieving from the second data source.

[0087] The second data source includes, but is not limited to, external search engines, web browsers, and news databases, and external resources other than those in the local knowledge base can be obtained through access to the second data source.

[0088] Specifically, a typical RAG system is relatively simple, mainly consisting of a retrieval unit (responsible for quickly finding a set of candidate documents relevant to the user's query from a large amount of text data, usually achieved through keyword matching or other similarity algorithms), a sorter (reordering the candidate documents returned by the retrieval unit to determine which documents are most likely to answer the user's query. This can be done using machine learning models, such as using deep neural networks to predict document relevance scores), and a language model. It does not involve the use of external search engines; that is, it first retrieves documents from a repository, then sorts them, and finally uses a language model to generate the answer.

[0089] In the embodiments of this specification, in addition to the local knowledge base (first data source), external search engines and news databases are also integrated (second data source). Thus, when searching based on a target query, it not only includes searching the local knowledge base, but also involves the use of external search engines and searching news databases. Second search results are obtained from the second data source according to the target query. By obtaining information from more and wider data sources, a more comprehensive and accurate answer is provided.

[0090] The data processing method provided in the embodiments of this specification integrates multiple data sources, enabling the system to synthesize information from different sources, reduce information silos, and improve the diversity and reliability of information.

[0091] Step 208: Based on the target query and the prompt text, obtain the target query result corresponding to the current query.

[0092] In one or more embodiments of this specification, by inputting the target query and prompt text into a language model, more accurate target query results are obtained. These results include not only the target answer to the current query but also related queries. Specific implementation methods are described below:

[0093] The step of obtaining the target query result corresponding to the current query based on the target query and the prompt text includes:

[0094] The target query and the prompt text are input into the language model to obtain the target answer corresponding to the current query and related queries related to the current query, wherein the related queries are queries that are associated with the current query.

[0095] The language model can be understood as LLM, or Large Language Model, which can generate coherent human language text and adjust its expression according to the context. In this process, LLM will refer to the prompt text to generate the final answer or response.

[0096] Specifically, the first and second search results obtained from different data sources are used to generate prompt text. The target query and prompt text are then input into the LLM, so that when the LLM generates the target query results, it can refer to the prompt text related to the target query to obtain more accurate target query results.

[0097] Furthermore, in practical applications, the target query results include the target answer and the related queries corresponding to the current query. These related queries are other query statements that are related to or may be triggered by the target query. Related queries can help users further explore or deepen their understanding of a certain topic.

[0098] The data processing method provided in the embodiments of this specification utilizes a language model to process target queries and prompt text, generating target answers and related queries. The prompt text can improve the quality of the target answer. Through a multi-stage information processing flow (multiple rounds of rewriting, retrieval, evaluation, and generation), it is ensured that the quality of the target answer can be effectively improved at each step.

[0099] In one or more embodiments of this specification, a user can click on the related queries displayed on the client user interface to identify the current query and the target answer as historical query data, and identify the related queries as the current query, thereby executing the above data processing steps. Specific implementation methods are as follows:

[0100] After obtaining the target answer corresponding to the current query and the related queries associated with the current query, the process further includes:

[0101] In response to the client's query request for the related query, the current query and the target answer are identified as the historical query data, the related query is identified as the current query, and the steps of identifying the current query and rewriting the current query based on the historical query data corresponding to the current query are executed to obtain the target query.

[0102] In practical applications, after obtaining the target query result corresponding to the current query based on the target query and the prompt text, the process further includes:

[0103] The target query result is sent to the client so that it can be displayed on the user interface of the client.

[0104] That is, after obtaining the target answer and related queries for the current query, the target answer and related queries for the current query are sent to the client, and the target answer and related queries for the current query are displayed on the user interface of the client.

[0105] Furthermore, users can click on the related queries displayed on the client's user interface to identify the current query and the target answer as historical query data, and identify the related queries as the current query, and continue to execute the above data processing methods.

[0106] The data processing method provided in the embodiments of this specification helps users discover more related information or topics by generating associated queries, broadening users' horizons and knowledge, and improving user experience.

[0107] The data processing method provided in the embodiments of this specification can rewrite the current query based on the historical query data corresponding to the current query, thereby obtaining a target query that is closer to the user's true intent. For the target, a first search result can be retrieved from a first data source, and the retrieved first search result can be evaluated to obtain an evaluation result of the first search result. When the target search strategy can be determined based on the evaluation result, more accurate retrieval can be achieved according to different search strategies, obtaining a first search result that is more relevant to the target query. Furthermore, a second search result can be obtained from a second data source. By obtaining search results from multiple data sources, prompt text is generated based on the first search result and the second search result obtained from the first data source and the second data source. Through the target query and the related prompt text, a more accurate target query result can be obtained.

[0108] Referring to Figure 3, Figure 3 shows a schematic diagram of the processing procedure of a data processing method provided in one embodiment of this specification.

[0109] Specifically, data processing methods are applied to data processing systems, which can be used in fields such as chatbots and question-answering systems to help people obtain the information they need faster and more accurately. In practical applications, a data processing system can be understood as a search and question-answering agent (intelligent agent).

[0110] This search and answer agent not only integrates information from local knowledge bases but also seamlessly connects to the powerful search capabilities of external search engines and further extends to real-time tracking of news databases, thereby ensuring that it can accurately capture the content that users need from a massive ocean of information.

[0111] Specifically, the user inputs the current query through the user interface of the client. The current query can be a precise search based on keywords or a fuzzy query expressed in natural language. After receiving the user's current query, the data processing system does not immediately perform a search. Instead, it first uses historical interaction data (i.e., historical query data in the above embodiment) to conduct in-depth analysis of the current query. This can include processing such as replacing some pronouns in the current query, thereby rewriting the current query into a target query. Based on the target query, the system understands the true intent behind the user's query (i.e., the target intent in the above embodiment).

[0112] By rewriting the current query multiple times, the rewritten target query not only becomes closer to the user's real needs, but also broadens the scope of the search and improves the diversity of search results by introducing synonyms, near-synonyms, or related concepts.

[0113] Based on the rewritten target query, the target intent is further extracted from the target query. Through the target intent, it can be ensured that the system always focuses on the content that users care about most when conducting information retrieval. After extracting the target intent, the system begins to conduct information retrieval. The retrieval process not only involves in-depth mining of the local knowledge base, but also makes full use of external search engines.

[0114] Specifically, based on the target intent, the system can first retrieve the first search result from the local knowledge base (there are multiple local knowledge bases, such as local knowledge base A, local knowledge base B, etc.). Then, the Critic component (an evaluation component used to assess or judge the relevance and quality of the search results and decide whether to continue searching or modify the query) is used to evaluate and judge the first search result. If the first search result is determined not to meet expectations, the current query can be rewritten in multiple rounds. If it is determined to continue searching based on the first search result, the rewritten target query is sent to multiple search engines (such as browser 1, browser 2) and news databases to obtain the second search result. That is, the system can quickly obtain the first search result and the second search result from different data sources.

[0115] The first and second search results can be used to customize prompts to facilitate the understanding and processing of the LLM. The generated prompts and the target query are input into the LLM (i.e., the language model in the above embodiment) to obtain the target answer corresponding to the current query and related questions (i.e., the associated queries in the above embodiment). By displaying the related questions to the user, the user can further understand similar or deeper questions based on the current query, thereby improving the user experience.

[0116] The data processing method provided in the embodiments of this specification, the search question-answering agent takes into account more factors, such as historical interactions, external resources and dynamic query rewriting. Through its highly intelligent interaction process and comprehensive and detailed information retrieval technology, it ensures that each step can effectively improve the quality of the final answer, thereby meeting the user's needs in an efficient and accurate way and improving the user experience.

[0117] Corresponding to the above method embodiments, this specification also provides a data processing system embodiment. Figure 4 shows a schematic diagram of the structure of a data processing system provided in one embodiment of this specification. As shown in Figure 4, it includes a rewriting unit 402, an evaluation unit 404, a generation unit 406, and an acquisition unit 408, wherein...

[0118] The rewriting unit 402 is used to determine the current query and rewrite the current query based on the historical query data corresponding to the current query to obtain the target query;

[0119] The evaluation unit 404 is used to retrieve a first search result from a first data source based on the target query, and to determine a target search strategy based on the evaluation result obtained by evaluating the first search result.

[0120] The generation unit 406 is used to retrieve a second search result from a second data source according to the target search strategy and the target query, and to generate a prompt text according to the first search result and the second search result;

[0121] The obtaining unit 408 is used to obtain the target query result corresponding to the current query based on the target query and the prompt text.

[0122] Optionally, the evaluation unit 404 includes an intent determination module, a first search module, and an evaluation module, wherein,

[0123] The intent determination module is used to determine the target intent corresponding to the target query based on the target query.

[0124] The first search module is used to retrieve a first search result from a first data source based on the target intent;

[0125] The evaluation module is used to determine the target search strategy based on the evaluation results obtained by evaluating the first search results.

[0126] Optionally, the evaluation module is further used to,

[0127] The relevance and quality of the first search result are evaluated to obtain the evaluation result, and the modified query strategy or the continued search strategy is determined based on the evaluation result.

[0128] Optionally, the generation unit 406 includes a second search module, wherein,

[0129] The second search module is used to retrieve the second search result from the second data source according to the continued search strategy and the target query.

[0130] Optionally, the generation unit 406 is further configured to:

[0131] If the target search strategy is determined to be the modified query strategy, the current query is rewritten based on the historical query data corresponding to the current query to obtain the modified query.

[0132] The modified query is identified as the target query, and the step of retrieving the first search result from the first data source based on the target query is continued until the target search strategy is determined to be the continuing search strategy based on the evaluation result obtained by evaluating the first search result. If the target search strategy is determined to be the continuing search strategy, the second search result is obtained from the second data source based on the target query.

[0133] Optionally, the obtaining unit 408 is further configured to,

[0134] Input the target query and the prompt text into the language model to obtain the target answer corresponding to the current query and related queries related to the current query.

[0135] The device further includes:

[0136] The determination module is used to respond to the client's query request for the related query, determine the current query and the target answer as the historical query data, determine the related query as the current query, execute the steps of determining the current query, rewriting the current query based on the historical query data corresponding to the current query, and obtaining the target query.

[0137] Optionally, the rewriting unit 402 is further configured to,

[0138] Receive the current query sent by the client, wherein the current query is determined by the client based on interactive operations on the user interface.

[0139] The device further includes:

[0140] The sending module is used to send the target query result to the client so that the target query result can be displayed on the user interface of the client.

[0141] The data processing system provided in one embodiment of this specification, through different units, can rewrite the current query based on historical query data corresponding to the current query, thereby obtaining a target query that is closer to the user's true intent. For the target, a first search result can be retrieved from a first data source, and the retrieved first search result can be evaluated to obtain an evaluation result. If a target search strategy can be determined based on the evaluation result, a more accurate search can be achieved according to different search strategies, obtaining a first search result more relevant to the target query. Furthermore, a second search result can be obtained from a second data source. By obtaining search results from multiple data sources, and generating prompt text based on the first and second search results obtained from the first and second data sources, a more accurate target query result can be obtained through the target query and related prompt text.

[0142] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0143] Figure 5 shows a structural block diagram of a computing device 500 according to one embodiment of this specification. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0144] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0145] In one embodiment of this specification, the aforementioned components of the computing device 500, as well as other components not shown in FIG. 5, may be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG. 5 is merely illustrative and not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0146] Computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). Computing device 500 can also be a mobile or stationary server.

[0147] The processor 520 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0148] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0149] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0150] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0151] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0152] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0153] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0154] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0155] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0156] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0157] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Determine the current query, and rewrite the current query based on the historical query data corresponding to the current query to obtain the target query; Based on the target query, a first search result is obtained by retrieving from the first data source, and a target search strategy is determined based on the evaluation result obtained by evaluating the first search result. Based on the target search strategy and the target query, a second search result is obtained from the second data source, and a prompt text is generated based on the first search result and the second search result; Based on the target query and the prompt text, obtain the target query result corresponding to the current query.

2. The data processing method according to claim 1, wherein the step of retrieving a first search result from a first data source based on the target query, and determining a target search strategy based on an evaluation result obtained by evaluating the first search result, comprises: Based on the target query, determine the target intent corresponding to the target query; Based on the stated target intent, a first search result is obtained by retrieving from a first data source, and a target search strategy is determined based on the evaluation result obtained by evaluating the first search result.

3. The data processing method according to claim 1 or 2, wherein the target search strategy includes modifying the query strategy or continuing the search strategy; The step of determining the target search strategy based on the evaluation result obtained by evaluating the first search result includes: The relevance and quality of the first search result are evaluated to obtain the evaluation result, and the modified query strategy or the continued search strategy is determined based on the evaluation result.

4. The data processing method according to claim 1 or 2, wherein the target search strategy includes modifying the query strategy or continuing the search strategy; The step of obtaining a second search result from a second data source based on the target search strategy and the target query includes: If the target search strategy is determined to be the modified query strategy, the current query is rewritten based on the historical query data corresponding to the current query to obtain the modified query. The modified query is identified as the target query, and the step of retrieving the first search result from the first data source based on the target query is continued until the target search strategy is determined to be the continuing search strategy based on the evaluation result obtained by evaluating the first search result. If the target search strategy is determined to be the continuing search strategy, the second search result is obtained from the second data source based on the target query.

5. The data processing method according to claim 1 or 2, wherein the target search strategy includes a continued search strategy; Based on the target search strategy and the target query, a second search result is obtained from the second data source, including: Based on the continued search strategy and the target query, the second search result is obtained by retrieving from the second data source.

6. The data processing method according to claim 1, wherein obtaining the target query result corresponding to the current query based on the target query and the prompt text includes: The target query and the prompt text are input into the language model to obtain the target answer corresponding to the current query and related queries related to the current query, wherein the related queries are queries that are associated with the current query.

7. The data processing method according to claim 6, after inputting the target query and the prompt text into a language model to obtain the target answer corresponding to the current query and related queries associated with the current query, further includes: In response to the client's query request for the related query, the current query and the target answer are identified as the historical query data, the related query is identified as the current query, and the steps of identifying the current query and rewriting the current query based on the historical query data corresponding to the current query are executed to obtain the target query.

8. The data processing method according to any one of claims 1 to 7, wherein determining the current query includes: Receive the current query sent by the client, wherein the current query is determined by the client based on interactive operations on the user interface; After obtaining the target query result corresponding to the current query based on the target query and the prompt text, the process further includes: The target query result is sent to the client so that it can be displayed on the user interface of the client.

9. The data processing method according to any one of claims 1 to 8, wherein rewriting the current query based on historical query data corresponding to the current query to obtain the target query includes: Identify pronouns or omitted expressions in the current query, and semantically complete the pronouns or omitted expressions based on the context information in the historical query data to generate the semantically complete target query.

10. The data processing method according to any one of claims 1 to 9, wherein the first data source includes a local knowledge base, and the second data source includes at least one of an external search engine and a news database; in, The external search engine includes a general-purpose search engine based on web page indexing, and the news library includes a real-time updated news information database.

11. The data processing method according to any one of claims 2 to 10, wherein the step of determining the target search strategy based on the evaluation result obtained by evaluating the first retrieval result is executed by a pre-trained evaluation model or a Critic component; The evaluation model scores the first search result based on at least one of the following: relevance, accuracy, completeness, and timeliness between the first search result and the target intent, and outputs the evaluation result.

12. The data processing method according to any one of claims 1 to 11, wherein generating prompt text based on the first search result and the second search result includes: The first and second search results are deduplicated, sorted, and abstracted to generate structured prompt text, which is then used as the context input for the language model.

13. The data processing method according to any one of claims 1 to 12, wherein the client includes at least one of a mobile application, a web browser, a mini-program, and a voice assistant; The user interface supports text input and / or voice input, and in the case of voice input, it also includes a speech recognition module that converts speech into text.

14. The data processing method according to any one of claims 7 to 13, further comprising, after determining the associated query as the current query: The historical query data is updated to include the dialogue history of the previous round's current query, target answer, and related queries. The new current query is then rewritten with context awareness based on the updated historical query data to achieve intent tracking and consistency maintenance in multi-round dialogues.

15. A data processing system, comprising a rewriting unit, an evaluation unit, a generation unit, and an acquisition unit, wherein, The rewriting unit is used to determine the current query and rewrite the current query based on the historical query data corresponding to the current query to obtain the target query; The evaluation unit is used to retrieve a first search result from a first data source based on the target query, and to determine a target search strategy based on the evaluation result obtained by evaluating the first search result. The generation unit is configured to retrieve a second search result from a second data source based on the target search strategy and the target query, and generate a prompt text based on the first search result and the second search result; The obtaining unit is used to obtain the target query result corresponding to the current query based on the target query and the prompt text.

16. The data processing system according to claim 9, wherein the evaluation unit comprises an intent determination module, a first search module, and an evaluation module, wherein, The intent determination module is used to determine the target intent corresponding to the target query based on the target query. The first search module is used to retrieve a first search result from a first data source based on the target intent; The evaluation module is used to determine the target search strategy based on the evaluation results obtained by evaluating the first search results.

17. The data processing system according to claim 9, wherein the generation unit includes a second search module, wherein, The second search module is used to retrieve the second search result from the second data source according to the continued search strategy and the target query.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 8.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 8.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 8.

Images