Model evaluation method, device, storage medium, and program product

Abstract

Embodiments of the present disclosure provide a model evaluation method, a device, a storage medium, and a program product. In the model evaluation method, an acquired query test case comprises a question-answer pair and a candidate knowledge document associated therewith, and a difficulty value of the query test case can be acquired on the basis of the candidate knowledge document. If the difficulty value of the query test case is less than or equal to a set difficulty threshold, the query test case can be evolved on the basis of an entity associated with the query test case, and a retrieval-augmented model to be evaluated is evaluated on the basis of the evolved query test case, to obtain an evaluation result. The candidate knowledge document is determined on the basis of the entity associated with the question-answer pair, and the entity is a relatively universal document attribute, so that the candidate knowledge document associated with the question-answer pair can be mined on the basis of different structures and different types of knowledge documents of the entity, thereby improving the universality of the evaluation method, and enabling the evaluation method to be flexibly migrated to different corpora.

Description

Model evaluation methods, equipment, storage media and software products Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model evaluation method, device, storage medium, and program product. Background Technology

[0002] Retrieval-Augmented Generation (RAG) is a technical framework that integrates information retrieval and natural language generation. It aims to improve the performance of language models in tasks such as question answering, text summarization, and dialogue systems by incorporating information from external knowledge bases. Before processing tasks, RAG models retrieve relevant domain-specific knowledge from external knowledge sources and then combine this retrieved knowledge with the language model's own understanding capabilities to generate more accurate and context-sensitive responses. In some application scenarios, it is necessary to construct effective benchmarks to evaluate the performance of RAG models. However, existing solutions suffer from poor transferability when evaluating RAG models. Therefore, a new solution is needed. Summary of the Invention

[0003] This disclosure provides a model evaluation method, apparatus, storage medium, and program product to improve the portability of the model evaluation method.

[0004] This disclosure provides a model evaluation method, comprising: obtaining a first query test case, the first query test case including: a first question-answer pair and its associated candidate knowledge documents, the candidate knowledge documents being determined based on entities associated with the first question-answer pair; obtaining a difficulty value of the first query test case based on the candidate knowledge documents; if the difficulty value of the first query test case is less than or equal to a set difficulty value threshold, then transforming the first query test case based on the entities associated with the first query test case to obtain a second query test case; and evaluating a retrieval-enhanced model under test based on the second query test case to obtain an evaluation result.

[0005] Optionally, obtaining the first query test case includes: obtaining at least one knowledge document from a knowledge base; generating a first question-and-answer pair by calling a content generation model based on the at least one knowledge document, the first question-and-answer pair including: a first query and its corresponding first reference answer result; obtaining the entity contained in the first query and the entity contained in the first reference answer result to obtain a first entity associated with the first question-and-answer pair; filtering out the candidate knowledge documents in the knowledge base based on the first entity; and using the multivariate data group composed of the first question-and-answer pair and the candidate knowledge documents as the first query test case.

[0006] Optionally, the first question-answer pair includes a first query and its corresponding first reference answer result; obtaining the difficulty value of the first query test case based on the candidate knowledge document includes: obtaining the key document required to generate the first reference answer result from the candidate knowledge document; obtaining the difficulty value of the first query test case based on the key document and / or the candidate knowledge document.

[0007] Optionally, obtaining the difficulty value of the first query test case based on the key document and the candidate knowledge document includes: taking the key document as the expected search result; obtaining the search difficulty value of the key document retrieved according to the first query based on the difference between the candidate knowledge document and the key document; and obtaining the difficulty value of the first query test case based on the search difficulty value.

[0008] Optionally, based on the differences between the candidate knowledge documents and the key documents, obtaining the retrieval difficulty value of the key document retrieved according to the first query includes: obtaining a first similarity between the first query and the candidate knowledge documents and a second similarity between the first query and the key document; and obtaining the retrieval difficulty value of the key document retrieved according to the first query based on the difference between the first similarity and the second similarity.

[0009] Optionally, it further includes: obtaining key sentences from the key documents on which the first reference response result depends; obtaining the difficulty value of the first query test case based on the retrieval difficulty value, including: obtaining the integration difficulty value of integrating the key sentences to obtain the first reference response result based on the key sentences and the key documents; obtaining the difficulty value of the first query test case based on the retrieval difficulty value and the integration difficulty value.

[0010] Optionally, obtaining the difficulty value of the first query test case based on the key document includes: obtaining the key sentence from the key document on which the generation of the first reference response result depends; obtaining the integration difficulty value of integrating the key sentence to obtain the first reference response result based on the key sentence and the key document; and obtaining the difficulty value of the first query test case based on the integration difficulty value.

[0011] Optionally, based on the key sentence and the key document, obtaining the integration difficulty value for integrating the key sentence to obtain the first reference response result includes: obtaining the information entropy corresponding to the key sentence based on the probability distribution of the key sentence in the key document; and obtaining the integration difficulty value for integrating the key sentence to obtain the first reference response result based on the information entropy corresponding to the key sentence.

[0012] Optionally, transforming the first query test case according to the entity associated with the first query test case to obtain a second query test case includes: obtaining the entities contained in the candidate knowledge document; obtaining the associated document of the candidate knowledge document according to the entities contained in the candidate knowledge document and the association relationship between multiple knowledge documents in the knowledge base, wherein the association relationship between the multiple knowledge documents is determined according to the entities contained in each of the multiple knowledge documents; obtaining a third query test case associated with the first query test case according to the associated document, wherein the third query test case contains a third question-answer pair, wherein the third question-answer pair includes: the third query and its corresponding third reference answer result; and transforming the first query test case according to the third query test case to obtain the second query test case.

[0013] Optionally, transforming the first query test case according to the third query test case to obtain the second query test case includes: transforming the first query according to the distribution characteristics of entities in the first question-answer pair and the third question-answer pair to obtain the transformed second query; calling a content generation model to generate the second query and its corresponding transformed second reference response result; obtaining the entities contained in the second query and the entities contained in the second reference response result to obtain the entities associated with the transformed second question-answer pair; filtering out the transformed candidate knowledge documents in the knowledge base according to the entities associated with the second question-answer pair; and using the multivariate data group composed of the second question-answer pair and the transformed candidate knowledge documents as the second query test case.

[0014] Optionally, based on the distribution characteristics of entities in the first question-and-answer pair and the third question-and-answer pair, the first query is transformed to obtain a transformed query, including: if an entity in the first query is the third reference response result, then the first query is inserted into the third query to obtain the transformed query; or, if the semantic relationship between an entity in the first reference response result and the entity corresponding to the third reference response result satisfies a set condition, then the first query and the third query are integrated to obtain the transformed query; or, if an entity in the first query is located in the third query, then the first query and the third query are concatenated to obtain the transformed query; or, if an entity in the first reference response result is located in the third reference response result, then a query with the first reference response result and the third reference response result as the response result is generated as the transformed query.

[0015] Optionally, the second query test case includes: a second query, a second reference response result, and a transformed candidate knowledge document; it also includes: calling a content generation model to obtain key information required to respond to the second query, and obtaining a transformed key document required to generate the second reference response result from the transformed candidate knowledge document; evaluating the retrieval-enhanced model under test based on the second query test case to obtain an evaluation result, including: inputting the second query and the transformed candidate knowledge document into the model under test to obtain a target knowledge document obtained by the model under test from the transformed candidate knowledge document based on the second query, and a target response result obtained by responding to the second query based on the target knowledge document; determining the knowledge retrieval capability score of the model under test based on the difference between the transformed key document and the target knowledge document; and determining the content generation capability score of the model under test based on the matching degree between the target response result and the key information; and obtaining the evaluation result based on the knowledge retrieval capability score and the content generation capability score.

[0016] This disclosure also provides a method for generating test cases for a retrieval enhancement model, comprising: obtaining a first query test case, the first query test case including: a first question-answer pair and its associated candidate knowledge documents, the candidate knowledge documents being determined based on entities associated with the first question-answer pair; obtaining a difficulty value of the first query test case based on the candidate knowledge documents; if the difficulty value of the first query test case is less than or equal to a set difficulty value threshold, then transforming the first query test case based on the entities associated with the first query test case to obtain a second query test case, the second query test case being used to evaluate a retrieval enhancement-based test model.

[0017] This disclosure also provides an electronic device, including: a memory and a processor; the memory is configured to store one or more computer instructions; the processor is configured to execute the one or more computer instructions to perform the steps in the method provided in this disclosure.

[0018] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of the method provided in this disclosure.

[0019] This disclosure also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in this disclosure.

[0020] In this embodiment, the obtained query test cases include question-answer pairs and their associated candidate knowledge documents. The difficulty value of the query test cases can be obtained based on the candidate knowledge documents. If the difficulty value of a query test case is less than or equal to a set difficulty threshold, the query test case can be transformed based on the entities associated with it. The transformed query test case is then used to evaluate the retrieval-enhanced model under test, yielding evaluation results. The candidate knowledge documents are determined based on the entities associated with the question-answer pairs. Entities are a relatively universal document attribute, facilitating the mining of candidate knowledge documents associated with question-answer pairs based on different structures and types of knowledge documents, thus improving the versatility of the evaluation method and allowing it to be flexibly transferred to different corpora. Furthermore, transforming the query test cases based on the entities associated with them enables dynamic adjustment of the evaluation difficulty value, allowing the evaluation results to more accurately reflect the capabilities of the model under test. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0022] Figure 1 is a flowchart illustrating a model evaluation method provided in an exemplary embodiment of this disclosure;

[0023] Figure 2 is a schematic diagram illustrating the evolution of query test cases provided in an exemplary embodiment of this disclosure;

[0024] Figure 3 is a schematic diagram of entity extraction based on corpus and establishment of association relationships between knowledge documents provided in an exemplary embodiment of this disclosure;

[0025] Figure 4 is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0027] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. The singular forms “a,” “the,” and “the” used in the embodiments of this disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0028] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0029] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0030] In the field of artificial intelligence technology, a benchmark refers to a set of standardized test cases, datasets, and performance metrics. Building an effective benchmark to evaluate the performance of a RAG model is a significant challenge.

[0031] Some benchmark generation schemes construct benchmarks by building multi-hop queries and knowledge conflict queries within a specific corpus. Multi-hop queries require extracting information from multiple documents or information sources to provide an answer, while knowledge conflict queries utilize information differences between new and old corpora. This benchmark generation method has certain limitations. Firstly, during model pre-training, the model may have already been pre-trained on a dataset containing a specific corpus. If this specific corpus is used to evaluate the pre-trained model, it may lead to "data leakage," meaning the model has already perceived or partially perceived the data in that specific corpus, resulting in evaluation results that do not reflect the model's true capabilities. Secondly, the content in a specific corpus usually has a defined type or structure and cannot fully represent other types of text, making this benchmark generation method difficult to transfer to other corpora.

[0032] In other approaches, schemas or relations can be extracted from knowledge documents, and queries can be automatically generated based on these schemas or relations. A corpus schema refers to a set of rules or formats used to organize and describe the structure of a corpus, primarily defining the organization of data, document structure, document annotation standards, and metadata attributes. However, this method cannot define the query difficulty value, thus hindering dynamic updates to the query difficulty. Furthermore, corpus schemas are a relatively rare attribute; when some corpora lack schema attributes, schema-based query generation methods cannot be executed on these corpora, which limits the scalability and portability of the baseline generation method to some extent.

[0033] To address the aforementioned technical problems, a solution is provided in some embodiments of this disclosure. The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0034] Figure 1 is a flowchart illustrating a model evaluation method provided in an exemplary embodiment of this disclosure. The method may include the steps shown in Figure 1:

[0035] Step 101: Obtain the first query test case, which includes: a first question-answer pair and its associated candidate knowledge documents, wherein the candidate knowledge documents are determined based on the entities associated with the first question-answer pair.

[0036] Step 102: Obtain the difficulty value of the first query test case based on the candidate knowledge document.

[0037] Step 103: If the difficulty value of the first query test case is less than or equal to the set difficulty value threshold, then the first query test case is transformed according to the entity associated with the first query test case to obtain the second query test case.

[0038] Step 104: Based on the second query test case, evaluate the model to be tested based on retrieval enhancement and obtain the evaluation results.

[0039] In step 101, the query test case is a benchmark used to test the model under test. The model under test can be any RAG model trained using artificial intelligence technology, such as an image recognition model, sentiment classification model, intelligent question answering model, speech recognition model, etc., based on RAG. This embodiment does not impose any restrictions. The model under test is used to perform the task of answering knowledge-intensive queries. This knowledge-intensive query refers to a query type that requires a large amount of background knowledge, external knowledge bases, or contextual information to obtain and understand the query results.

[0040] Query test cases are used to test the ability of the model under test to retrieve knowledge based on a query and respond to the query based on the retrieved knowledge. Each query test case may include at least a candidate knowledge document R and a question-answer pair consisting of a query Q and a reference response A; that is, a query test case can be described as a multivariate dataset {Q, R, A}. The candidate knowledge document is used to evaluate the model under test's ability to retrieve knowledge based on a query, and the reference response is a desired response used to evaluate the model under test's ability to respond to the query based on the knowledge retrieved from the candidate knowledge document.

[0041] The first query test case refers to any query test case. The term "first" is used here to describe the query test case only to facilitate the distinction between the query test case before and after the transformation, and does not impose any restrictions on the order or position of the query test cases.

[0042] Knowledge documents are structured or unstructured text data, including but not limited to documents, press releases, web page content, e-books, and social media posts. The first query test case includes the first question-answer pair and candidate knowledge documents; the first question-answer pair includes the first query and its corresponding first reference response.

[0043] In some optional embodiments, candidate knowledge documents are selected from the knowledge base based on the entities contained in the first question-and-answer pair. An entity refers to a specific object or concept with independent meaning; it can be a concrete thing such as a person's name, place name, organization, time, currency amount, percentage, or an abstract concept. The entities contained in the first question-and-answer pair may include the entities contained in the first query and / or the entities contained in the first reference response result; this embodiment is not limited to this. In some optional embodiments, candidate knowledge documents can be selected from the knowledge base based on the entities contained in the first query and the entities contained in the first reference response result. The entities contained in the first query and the entities contained in the first reference response result can cover as many entities as possible involved in the first query, thereby increasing the knowledge capacity of the candidate knowledge documents.

[0044] Optionally, when obtaining the first query test case, at least one knowledge document can be retrieved from the knowledge base, and a content generation model can be invoked to generate a first question-and-answer pair based on the at least one knowledge document. The first question-and-answer pair includes a first query and its corresponding first reference answer result. The content generation model can be a trained and fine-tuned model that demonstrates good performance on content generation tasks. In this implementation, prompt words can be constructed based on the at least one knowledge document, and the task of generating the first query and the first reference answer result based on the at least one knowledge document can be configured in the prompt words. This allows the content generation model to utilize its learned text understanding and text generation capabilities to generate the first query and the first reference answer result under the guidance of the prompt words.

[0045] The content generation model can be a large language model, which is a naturally language processing (NLP) model trained on a large scale. Large language models are typically built on deep learning techniques and trained on large training datasets, enabling them to exhibit powerful performance in handling NLP tasks. Given a text (i.e., context), the large language model attempts to predict the most likely next word. The number of parameters in a large language model is greater than a set threshold, typically in the millions or billions. During training, these parameters are continuously adjusted and optimized based on the difference between the large language model's predictions and actual results to improve its prediction accuracy. In some embodiments, large language models typically employ advanced neural network architectures, such as the Transformer architecture, to build their model structure. This architecture enables the large language model to capture complex patterns in the text and handle long-distance dependencies. After sufficient training, the large language model possesses powerful generative capabilities, producing coherent and context-appropriate text content based on given prompts.

[0046] In other alternative implementations, the first reference response result can be obtained by manually annotating the first query. During manual annotation, the first query and candidate knowledge documents can be displayed on the terminal device used by the annotation expert, and the result of the annotation expert's response to the first query based on the candidate knowledge documents can be obtained as the first reference response result.

[0047] After obtaining the first query and its corresponding first reference answer result, the entities contained in the first query and the entities contained in the first reference answer result can be obtained to obtain the first entity associated with the first question-answer pair. Based on the first entity, candidate knowledge documents are filtered in the knowledge base. The first entity can be one or more entities. It should be understood that in the above and following embodiments of this disclosure, the use of "first" and "second" to define the descriptive objects is used to distinguish different descriptive objects and is not used to limit the number or order of different descriptive objects; this will not be elaborated further.

[0048] Optionally, the entities contained in each knowledge document in the knowledge base can be pre-identified. Optionally, for any given knowledge document, the entities contained in that knowledge document can be identified based on rule-based matching methods. For example, regular expressions can be used to match specific formats in the text, such as dates, email addresses, or phone numbers. For example, a vocabulary containing known entities (such as names of people or places) can be obtained, and entities matching entries in the vocabulary can be queried in the knowledge document. Any knowledge document may include one or more entities, and the same entity may reside in different knowledge documents.

[0049] After obtaining the entities contained in each knowledge document in the knowledge base based on the above implementation method, candidate knowledge documents can be selected from the knowledge base based on the entity co-occurrence principle. The entity co-occurrence principle refers to the principle that the same or similar entities in the first entity appear in knowledge documents. Optionally, if any entity in the first entity appears in a certain knowledge document, that knowledge document can be added as a candidate knowledge document. Optionally, if a similar entity of any entity in the first entity appears in a certain knowledge document, that knowledge document can be added as a candidate knowledge document.

[0050] After obtaining the candidate knowledge documents, the multivariate dataset consisting of the first question-answer pair and the candidate knowledge documents can be used as the first query test case. The multivariate dataset corresponding to the first query test case can be represented as {Q=q1,R=r1,A=a1}.

[0051] Based on this implementation method, candidate knowledge documents related to the question-answer pairs can be mined from the knowledge base according to the entities contained in the question-answer pairs, thus facilitating the testing of the knowledge retrieval capabilities of the model under test. Entities are relatively universal attributes in knowledge documents; therefore, even if knowledge documents in the knowledge base have various different patterns, the relationship between the knowledge document and the question-answer pair can be determined based on the entities contained in the knowledge document, making it easy to transfer the model evaluation method to corpora in different domains. For example, when legal documents, medical instructions, and financial knowledge documents have different patterns, the model evaluation method can also be transferred to corpora related to legal documents, medical instructions, or financial knowledge.

[0052] After obtaining the first query test case, in step 102, the difficulty value of the first query test case can be obtained based on the candidate knowledge documents. The difficulty value of the query test case refers to the degree to which the query test case poses a challenge to the model under test, reflecting the extent to which the query test case can reveal the model's strengths and limitations. In this embodiment, the model under test is a model based on retrieval enhancement technology; therefore, the difficulty value of the first query test case is related to the degree to which the first query test case poses a challenge to the retrieval enhancement capability of the model under test. Retrieval enhancement capability refers to the ability to filter out key documents from the knowledge documents related to the query that are relied upon to answer the query, and to generate the corresponding response results based on the key documents. Based on this, in this embodiment, one or more difficulty value evaluation indicators can be constructed based on the candidate knowledge documents related to the query and the key documents relied upon to answer the query, and the difficulty value of the query test case can be obtained based on the evaluation indicators. The following will continue to use the first query test case as an example for illustrative explanation.

[0053] Optionally, when obtaining the difficulty value of the first query test case based on the candidate knowledge documents, the key documents required to generate the first reference response result can be obtained from the candidate knowledge documents. That is, the first query test case can be described as a multivariate data set {Q, R, GR, A} consisting of the first query Q, candidate knowledge documents R, key documents GR, and reference response result A. In some optional embodiments, the key documents can be obtained by a content generation model through knowledge retrieval based on the first query. This content generation model can be another model with higher task processing performance or more parameters than the model under test. The content generation model, after training and fine-tuning, has high knowledge retrieval capabilities and can be used to generate query test cases for the model under test. In other optional embodiments, the key documents can be obtained through manual knowledge annotation based on the first query. During manual knowledge annotation, the first query and candidate knowledge documents can be displayed on the terminal device used by the annotation expert, and the key documents required to respond to the first query can be obtained based on the annotation expert's filtering operation of the candidate knowledge documents. After obtaining the key documents, the difficulty value of the first query test case can be obtained based on the key documents and / or candidate knowledge documents.

[0054] In this implementation, constructing an index for evaluating the difficulty value of the first query test case based on candidate documents and key documents in the candidate documents required to answer the first query can make the index consistent with the operating mechanism of the model under test, thereby facilitating accurate evaluation of the true performance of the model under test.

[0055] In some optional embodiments A1, when obtaining the difficulty value of the first query test case based on the key document and candidate knowledge documents, the key document can be used as the expected search result, and the search difficulty value of the key document retrieved based on the first query can be obtained based on the difference between the candidate knowledge documents and the key document. The difference between the candidate knowledge documents and the key document reflects the deviation between the search scope corresponding to the first query and the expected search result. The greater the deviation, the greater the search difficulty value corresponding to the first query test case.

[0056] In some alternative embodiments, the overlap between candidate knowledge documents and key documents can be calculated, and this overlap can be used as the difference between the candidate knowledge documents and key documents. For example, Jaccard similarity between candidate knowledge documents and key documents can be calculated to determine the proportion of shared vocabulary between the candidate knowledge documents and key documents, and this proportion can be used as the overlap.

[0057] In some alternative embodiments, the difference between candidate knowledge documents and key documents can be calculated by combining the first query. Optionally, a first similarity between the first query and candidate knowledge documents and a second similarity between the first query and key documents can be obtained. The first and second similarities can be calculated using at least one of the following methods: edit distance-based similarity calculation, Jaccard similarity, cosine similarity, and deep learning models; details are omitted here. Based on the difference between the first and second similarities, the retrieval difficulty value of the key documents retrieved based on the first query is obtained. For example, the difference between the first and second similarities can be directly calculated as the retrieval difficulty value. Alternatively, the ratio of the first and second similarities can be calculated as the retrieval difficulty value; this embodiment is not limited in this respect. Optionally, when candidate knowledge documents include multiple documents, the first similarity can be the larger value among the similarities between the first query and multiple documents; when key documents include multiple documents, the first similarity can be the average of the similarities between the first query and multiple documents. The following will use Formula 1 as an example to illustrate one possible calculation method for the retrieval difficulty value.

[0058] Where RD(QRA) represents the retrieval difficulty value corresponding to query q. GRq represents the key document set, R represents the candidate knowledge documents obtained by the RAG model based on query q. r represents any candidate knowledge document in the candidate knowledge document set R, and sim(q,r) is the similarity between query q and candidate knowledge document r. r∈R (sim(q,r)) represents the maximum similarity between query q and key documents in the candidate knowledge document set R. |GRq| represents the size of the key document set GRq related to query q, where gri It is the i-th key document in the key document set GRq. This represents the sum of similarities between query q and each key document in the key document set GRq. Based on Equation 1, the difference between the retrieval results obtained by the RAG model based on query q and the expected retrieval results corresponding to query q can be obtained, thereby determining the difficulty value of the RAG model in retrieving knowledge documents related to query q.

[0059] After obtaining the search difficulty value based on the above implementation method, the difficulty value of the first query test case can be obtained according to the search difficulty value. Optionally, the search difficulty value can be directly used as the difficulty value of the first query test case, or the search difficulty value can be modified according to other types of difficulty values ​​to obtain the difficulty value of the first query test case.

[0060] In this implementation, by calculating the difference in similarity between the first query and the candidate knowledge documents and key documents, the deviation between the search scope corresponding to the first query and the expected search results can be accurately obtained, thereby accurately obtaining the search difficulty value corresponding to the first query test case.

[0061] In some optional embodiments A2, when obtaining the difficulty value of the first query test case based on the key document and candidate knowledge documents, the retrieval difficulty value and integration difficulty value of the first query test case can be obtained, and the retrieval difficulty value and integration difficulty value can be merged to obtain the difficulty value of the first test case. The integration difficulty value refers to the difficulty value of understanding and integrating some sentences in the key document to generate a response result.

[0062] Optionally, after obtaining the key document, key sentences dependent on the generation of the first reference response result can be extracted from the key document. These key sentences are extracted from the key document and contain information directly used to generate the first reference response result. In some embodiments, a content generation model can extract key sentences dependent on the generation of the first reference response result from the key document. The content generation model can use keyword matching to find sentences in the key document with high textual or semantic overlap with the first query and the first reference response result as key sentences. Alternatively, the content generation model can perform syntactic analysis on the sentences in the key document, identify the subject, verb, and object structures, and retrieve sentences conveying core information from the key document based on the identified sentence structures as key sentences. In other embodiments, expert annotation can be used to annotate the key sentences required to generate the first reference response result in the key document, which will not be elaborated further. After obtaining the key sentences, the first query test case can be described as a multivariate data set {Q, R, GR, KS, A} consisting of the first query Q, candidate document R, key document GR, key sentence KS, and reference response result A.

[0063] Based on key sentences and key documents, the integration difficulty value for integrating key sentences to obtain the first reference response result can be obtained.

[0064] Optionally, the information entropy corresponding to the key sentences can be obtained based on the probability distribution of key sentences in key documents, and the integration difficulty value for integrating the key sentences to obtain the first reference response result can be obtained based on the information entropy corresponding to the key sentences. Here, information entropy is used to quantify the degree of uncertainty or randomness of information. When the information entropy of multiple key sentences is high, it indicates that the key sentences are not easily identified, thus increasing the difficulty of integrating different key sentences to obtain the first reference response result. The following will use Formula 2 as an example to illustrate one optional calculation method for the integration difficulty value.

[0065] Where GD(QRA) represents the global dissimilarity measure of the RAG model. KSq is the set of key sentences required to answer query q. H(KSq) represents the information entropy of the knowledge state KSq of query q. i This represents the probability distribution of the i-th key sentence in the key document set GRq, which reflects the probability distribution of key sentence p. i Importance or relevance in KSq. H(KS) qFormula 2 is used to calculate the information entropy of the probability distribution of all key sentences in the key document set GRq. Higher information entropy indicates greater uncertainty regarding the key sentences. Based on Formula 2, the uncertainty or information content of the key sentences dependent on query q can be measured by calculating the information entropy of the probability distribution of the retrieval results. This can serve as a difficulty value for the RAG model to integrate knowledge documents related to query q.

[0066] Based on the retrieval difficulty value and the integration difficulty value, the difficulty value of the first query test case is obtained. In some optional embodiments, as shown in Figure 2, the retrieval difficulty value GD(QRA) and the integration difficulty value RD(QRA) can be weighted and summed according to a set weighting coefficient λ to obtain the total difficulty value RGD(QRA), as shown in the following formula 3: RGD(QRA)=GD(QRA)+λ-RD(QRA) Formula 3

[0067] Based on this implementation method, the difficulty value of integrating key sentences to generate response results can be obtained according to the probability distribution of key sentences in key documents, thereby more accurately evaluating the performance of the model under test.

[0068] In some optional embodiments A3, when obtaining the difficulty value of the first query test case based on the key document and candidate knowledge documents, the key sentences on which the generation of the first reference response result depends can be obtained from the key document. Based on the key sentences and the key document, the integration difficulty value for integrating the key sentences to obtain the first reference response result can be obtained, and the difficulty value of the first query test case can be obtained based on the integration difficulty value. In this implementation, the integration difficulty value can be directly used as the difficulty value of the first query test case. Optional implementation methods for obtaining the integration difficulty value can refer to the foregoing embodiments and the descriptions related to Formula 3, and will not be repeated here.

[0069] After obtaining the difficulty value of the first query test case based on the above implementation method, as shown in Figure 2, it can be determined whether the difficulty value of the first query test case is greater than a set difficulty threshold. If it is greater, the first query test case can be directly used as the query test case for evaluation. If it is determined that the difficulty value of the first query test case is less than or equal to the set difficulty threshold, in step 104, the first query test case can be transformed according to the entities associated with it to obtain the transformed second query test case. The transformation of the first query test case, also known as evolution, refers to improving, expanding, or deforming the first query test case to ensure that the transformed first query test case can more effectively evaluate the performance of the model under test and can add new challenges to the model under test.

[0070] The entities associated with the first test case may include at least one of the following: entities contained in the first query, entities contained in the first reference response result, and entities contained in the candidate knowledge documents. When transforming the first query test case based on the entities associated with it, the transformation can be based on the principle of entity co-occurrence, searching for other query test cases associated with the first query test case from the existing query test cases, and then transforming the first query test case according to the searched other query test cases. Alternatively, the transformation can be based on the principle of entity co-occurrence, searching for more knowledge documents associated with the first query test case from the knowledge base, and then transforming the first query test case according to the searched more knowledge documents. Optionally, transforming the first query test case may include: improving the first query, and / or improving the first reference response result, and / or expanding the candidate knowledge documents; this embodiment is not limited to these aspects. The following will provide exemplary descriptions in conjunction with different implementation methods.

[0071] In some optional embodiments B1, when transforming the first query test case based on the entity associated with it, a third query test case associated with the first query test case can be obtained, and the first query test case can be transformed based on the third query test case to obtain a second query test case. The third query test case is any other query test case besides the first query test case, and its generation method is the same as that of the first query test case, as described in the foregoing embodiments. The third query test case includes a third question-answer pair and candidate knowledge documents. The third question-answer pair includes a third query and its corresponding third reference response result.

[0072] Optionally, the third query test case is obtained based on the relationships between knowledge documents. As shown in Figure 2, the entities contained in the candidate knowledge document can be obtained, and the associated documents of the candidate knowledge document can be obtained based on the entities contained in the candidate knowledge document and the relationships between multiple knowledge documents in the knowledge base. Based on the associated documents, the third query test case associated with the first query test case can be obtained.

[0073] The relationships between multiple knowledge documents in the knowledge base are determined based on the entities contained in each of the multiple knowledge documents. The following, with reference to Figure 3, will provide an exemplary description of optional implementation methods for obtaining the relationships between multiple knowledge documents.

[0074] Optionally, as shown in Figure 3, a corpus collection operation can be performed to obtain a knowledge base, and entity extraction can be performed on multiple knowledge documents in the knowledge base to obtain the entities contained in each knowledge document. Based on the entities contained in each knowledge document and the entity co-occurrence principle, the relationships between the multiple knowledge documents are obtained. The entity co-occurrence principle refers to the principle that the same or similar entities appear in different knowledge documents. If the same entity appears in different knowledge documents, then these different knowledge documents can be considered to have a certain connection. Alternatively, if different entities appearing in different knowledge documents are similar or have a relationship, then these different knowledge documents can be considered to have a certain connection. Optionally, for any two knowledge documents among the multiple knowledge documents, if any two knowledge documents contain the same or similar entities, then these two knowledge documents are considered to have a relationship.

[0075] In some optional embodiments, in addition to the entity co-occurrence principle, if different knowledge documents have similar entity patterns, then a certain connection can be considered between these different knowledge documents. Here, an entity pattern refers to a specific combination, order, or contextual structure in which entities appear in a knowledge document. For example, an entity pattern might be: [Date], [Reporter's Name] reported that [Event Location] experienced [Event Type]. If the entity patterns in different knowledge documents are the same or similar, then a certain connection can be considered between these different knowledge documents.

[0076] Based on the above implementation method, valuable related information can be extracted from multiple knowledge documents according to the entities contained in the knowledge documents. Entities are relatively universal attributes in knowledge documents. Therefore, even when multiple knowledge documents are of different types, the relationship between knowledge documents of various patterns can be determined based on entities, which facilitates the transfer of model evaluation methods to corpora in different domains.

[0077] In some optional embodiments, after obtaining the relationships between multiple knowledge documents, an entity-document relationship graph corresponding to the multiple knowledge documents can be drawn based on these relationships. Optionally, as shown in Figure 3, documents can be connected to entities contained in the documents, thereby determining documents with related relationships based on the same entities, facilitating subsequent queries. Based on the above, in the process of obtaining the associated documents of the candidate knowledge document, for each entity contained in the candidate knowledge document, a knowledge document containing that entity can be searched in the knowledge base according to the relationships between multiple knowledge documents in the knowledge base, serving as an associated document of the candidate knowledge document. Based on the associated document, a third query test case associated with the first query test case can be obtained. Optionally, for any document in the associated documents, a query test case containing that document can be searched in the constructed query test cases, serving as a third query test case associated with the first query test case. Then, the first query test case can be transformed based on the third query test case to obtain a second query test case.

[0078] Optionally, in addition to obtaining the third query test case based on the association relationship of knowledge documents, the third query test case can also be obtained based on the associated entities of the first question and answer. Optionally, it can be determined whether any entity in the first entity or a similar entity of that entity appears in a certain query or whether it appears in the reference answer results corresponding to the query. If either of the above determination results is yes, then it is determined that the query test case corresponding to the query is associated with the first query test case.

[0079] After obtaining the third query test case based on the above implementation method, the first query test case can be transformed according to the third query test case to obtain the second query test case.

[0080] After obtaining the third query test case, the first query can be transformed based on the distribution characteristics of entities in the first and third question-and-answer pairs to obtain the transformed second query. This transformation includes the positions of identical or similar entities in the first and third question-and-answer pairs, and the relationship between these entities and the first and second question-and-answer pairs. The position of an entity in any question-and-answer pair can include: the entity appearing in the query of the question-and-answer pair, or the entity appearing in the answer of the question-and-answer pair. An entity appearing in the query means that the entity is directly part of the user's question; for example, if the entity is "France," and the query is: "What is the capital of France?", then the entity appears in the answer. An entity appearing in the answer means that the query may be broad or involve multiple aspects, while the answer provides specific entity information. For example, if the entities are "Leonardo da Vinci," "Michelangelo," and "Raphael," and the query is "List some Italian painters," the answer would be "Leonardo da Vinci, Michelangelo, and Raphael are three Italian painters of the Renaissance period."

[0081] The relationship between an entity and any question-and-answer pair can include: the entity is the query in the question-and-answer pair, or the entity is the answer in the question-and-answer pair. When the entity is the answer in the question-and-answer pair, the result of the query is itself a specific entity; that is, the purpose of the query is to obtain information about this entity. For example, the entity is "Paris," and the query is "What is the capital of France?".

[0082] When the distribution characteristics of the first entity are different, the transformation method for the first query will also be different. The following will illustrate the transformation method of the first query with reference to the different distribution characteristics shown in Figure 2.

[0083] Optionally, the distribution characteristics of the first entity in the first and third question-answer pairs may include: the entity in the first query is a third reference response result. Accordingly, based on this distribution characteristic, one way to transform the first query may be to insert the first query into the third query to obtain the transformed second query. Taking question-answer pairs (q1, a1) and (q2, a2) as examples, if the entity e in query q1 is a reference response result a2 corresponding to another query q2, i.e., e∈q1 and e=a2, then query q1 can be inserted into query q2 to obtain the transformed query q3.

[0084] Optionally, the distribution characteristics of the first entity in the first and third question-answer pairs may include: the semantic relationship between the entity in the first reference response result and the entity corresponding to the third reference response result satisfies a set condition. This set condition may be a semantic similarity greater than a specified threshold, which may be 80%, 85%, or other optional thresholds; this embodiment does not impose any limitations. Accordingly, one way to transform the first query based on this distribution characteristic to obtain the transformed query may include: integrating the first query and the third query to obtain the transformed second query. Taking question-answer pairs (q1, a1) and (q2, a2) as examples, if the semantic relationship between entity e1 in reference response result a1 and entity e2 corresponding to another reference response result a2 satisfies a set condition, i.e., e1∈a1 and e2=a2, then query q1 and then query q2 are integrated to obtain the transformed query q3. Here, integration refers to combining multiple queries using logical operators based on the semantics or structure of different queries. For example, q1 is: "What is the weather like in city C1 today?", q2 is: "What is the weather like in city C1 tomorrow?", and the combined q3 could be: "What is the weather like in city C1 for the next two days?".

[0085] Optionally, the distribution characteristics of the first entity in the first and third question-answer pairs may include: the entity in the first query is located in the third query. Accordingly, based on this distribution characteristic, one way to transform the first query may be to concatenate the first query with the third query to obtain the transformed second query. Taking question-answer pairs (q1, a1) and (q2, a2) as examples, if the entity e in query q1 is located in another query q2, i.e., e∈q1 and e∈q2, then concatenating query q1 with query q2 yields the transformed query q3. Optionally, this concatenation method may be to directly concatenate the first and third queries, ensuring that each query retains its independence after concatenation.

[0086] Optionally, the distribution characteristics of the first entity in the first and third question-answer pairs may include: the entity in the first reference response result is located in the third reference response result. Accordingly, based on this distribution characteristic, one way to transform the first query may include: generating a query with the first and third reference response results as response results, as the transformed second query. Taking question-answer pairs (q1, a1) and (q2, a2) as examples, if entity e in reference response result a1 is located in another reference response result a2, i.e., e∈a1 and e∈a2, then a new query with a2 and a2 as response results can be generated, as the transformed query q3. Optionally, in this implementation, a content generation model can be invoked to generate a query with the first and third reference response results as response results.

[0087] Based on the above implementation method, after transforming the first query to obtain the second query, the content generation model can be called to generate the second query and its corresponding transformed second reference response result; obtain the entities contained in the second query and the entities contained in the second reference response result to obtain the entities associated with the transformed second question-and-answer pair; based on the entities associated with the second question-and-answer pair, filter out the transformed candidate knowledge documents in the knowledge document base; and use the multivariate data group composed of the second question-and-answer pair and the transformed candidate knowledge documents as the second query test case.

[0088] Based on this implementation method, different queries can be transformed by the entities contained in different query test cases to obtain query test cases with different difficulty values. This realizes a scheme for dynamically adjusting query test cases based on difficulty value indicators, which is conducive to more comprehensive testing of the model under test.

[0089] In some optional embodiments B2, when transforming the first query test case based on the entities associated with it, other knowledge documents related to the candidate knowledge documents in the knowledge base can be searched based on the entities contained in the candidate knowledge documents in the first query test case. These other knowledge documents are then used to expand the candidate knowledge documents, increasing the diversity of candidate knowledge documents in the first query test case and thus increasing the difficulty of retrieving key documents from the expanded candidate knowledge documents. Optionally, when searching for other knowledge documents related to the candidate knowledge documents in the knowledge base, an entity-document relationship graph can be queried based on the entities contained in the candidate knowledge documents to obtain documents related to the candidate knowledge documents. These related documents are then added to the candidate knowledge documents to increase the diversity of candidate knowledge documents in the first query test case, thereby expanding the amount of knowledge retrieved.

[0090] After obtaining the second query test case based on the above implementation method, in step 104, the model under test based on retrieval enhancement can be evaluated according to the second query test case to obtain the evaluation result. The second query test case includes: a second query, a second reference response result, and a transformed candidate knowledge document. When evaluating the model under test, the second query and the transformed candidate knowledge document can be input into the evaluation model, and the output result of the evaluation model can be obtained. This result includes: the result of retrieving the transformed candidate knowledge document according to the second query, and the response result obtained by responding to the second query according to the retrieval result. Based on the retrieval result and the response result, the knowledge retrieval capability and enhancement generation capability of the model under test can be evaluated respectively.

[0091] In some optional embodiments, after obtaining the second query test case, it can be further determined whether the difficulty value of the second query test case is less than or equal to a set difficulty threshold. If yes, an operation to transform the second query test case can be performed, as described in the foregoing embodiments. If no, the test model based on retrieval enhancement can be evaluated based on the second query test case, which will not be elaborated further.

[0092] In some optional embodiments, a content generation model can be further invoked to obtain key information required to answer the second query, and transformed key documents required to generate the second reference response result can be obtained from the transformed candidate knowledge documents. Based on this, the second query test case can be described as a multivariate data set {Q, R, GR, KS, KP, A} consisting of the second query Q, the transformed candidate document R, the transformed key document GR, the key sentence KS on which the second reference response result depends, and the second reference response result A. In some embodiments, as shown in FIG3, the entities contained in the second query and the entity E contained in the second reference response result can be stored in this multivariate data set for subsequent queries.

[0093] Accordingly, when evaluating the retrieval-enhanced model based on the second query test case, the second query and the transformed candidate knowledge documents can be input into the model. This yields the target knowledge document retrieved by the model from the transformed candidate knowledge documents based on the second query, and the target response result obtained by the model responding to the second query based on the target knowledge document. Optionally, the knowledge retrieval capability score of the model can be determined based on the difference between the transformed key document and the target knowledge document. The greater the difference between the transformed key document and the target knowledge document, the greater the deviation between the retrieval result and the expected result, and the lower the knowledge retrieval capability of the model.

[0094] Optionally, the content generation capability score of the model under test can be determined based on the matching degree between the target response and the key information. In some implementations, the semantic similarity between the key information and the target response is calculated, and the content generation capability score of the model under test is determined based on the semantic similarity. The higher the semantic similarity between the key information and the target response, the more key information the target response covers, and the higher the content generation capability score of the model under test. In other implementations, the key information and the target response can be input into the content generation model, and prompt words can be constructed to guide the content generation model to calculate the matching degree of the key information, and output the content generation capability score of the model under test. The higher the matching degree between the target response and the key information.

[0095] After obtaining the knowledge retrieval ability score and content generation ability score of the model under test, the evaluation result of the model under test can be obtained based on the knowledge retrieval ability score and content generation ability score.

[0096] In this implementation, the acquired query test cases include question-answer pairs and their associated candidate knowledge documents. Based on the candidate knowledge documents, the difficulty value of the query test cases can be obtained. If the difficulty value of a query test case is less than or equal to a set difficulty threshold, the query test case can be transformed based on the entities associated with it. The transformed query test case is then used to evaluate the retrieval-enhanced model under test, yielding evaluation results. The candidate knowledge documents are determined based on the entities associated with the question-answer pairs. Entities are a relatively universal document attribute, facilitating the mining of candidate knowledge documents related to the question-answer pairs based on knowledge documents with different structures and patterns. This improves the versatility of the evaluation method, allowing it to be flexibly transferred to different corpora. Furthermore, transforming the query test cases based on the entities associated with them enables dynamic adjustment of the evaluation difficulty value, ensuring that the evaluation results more accurately reflect the capabilities of the model under test.

[0097] In addition to the foregoing embodiments, this disclosure also provides a method for generating test cases for a retrieval enhancement model, comprising: obtaining a first query test case, the first query test case including: a first question-answer pair and its associated candidate knowledge documents, the candidate knowledge documents being determined based on entities associated with the first question-answer pair; obtaining a difficulty value of the first query test case based on the candidate knowledge documents; if the difficulty value of the first query test case is less than or equal to a set difficulty value threshold, then transforming the first query test case based on the entities associated with the first query test case to obtain a second query test case, the second query test case being used to evaluate a retrieval enhancement-based model under test. For details, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0098] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0099] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0100] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0101] Figure 4 illustrates a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure, which is applicable to the model evaluation method provided in the foregoing embodiments. As shown in Figure 4, the electronic device includes a memory 401 and a processor 402.

[0102] Memory 401 is configured to store computer programs and may be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.

[0103] Processor 402, coupled to memory 401, is configured to execute a computer program in memory 401 for: obtaining a first query test case, the first query test case including: a first question-answer pair and its associated candidate knowledge documents, the candidate knowledge documents being determined based on entities associated with the first question-answer pair; obtaining a difficulty value of the first query test case based on the candidate knowledge documents; if the difficulty value of the first query test case is less than or equal to a set difficulty value threshold, transforming the first query test case based on the entities associated with the first query test case to obtain a second query test case; and evaluating a retrieval-enhanced model under test based on the second query test case to obtain an evaluation result.

[0104] Optionally, when the processor 402 obtains the first query test case, it is specifically configured to: obtain at least one knowledge document from the knowledge base; generate a first question-and-answer pair by calling a content generation model based on the at least one knowledge document, wherein the first question-and-answer pair includes: a first query and its corresponding first reference answer result; obtain the entity contained in the first query and the entity contained in the first reference answer result to obtain a first entity associated with the first question-and-answer pair; filter out the candidate knowledge documents in the knowledge base based on the first entity; and use the multivariate data group composed of the first question-and-answer pair and the candidate knowledge documents as the first query test case.

[0105] Optionally, the first question-answer pair includes a first query and its corresponding first reference answer result; when the processor 402 obtains the difficulty value of the first query test case based on the candidate knowledge document, it is specifically configured to: obtain the key document required to generate the first reference answer result from the candidate knowledge document; and obtain the difficulty value of the first query test case based on the key document and / or the candidate knowledge document.

[0106] Optionally, when the processor 402 obtains the difficulty value of the first query test case based on the key document and the candidate knowledge document, it is specifically configured to: use the key document as the expected search result; obtain the search difficulty value of the key document retrieved based on the difference between the candidate knowledge document and the key document; and obtain the difficulty value of the first query test case based on the search difficulty value.

[0107] Optionally, when the processor 402 obtains the retrieval difficulty value of the key document retrieved according to the first query based on the difference between the candidate knowledge document and the key document, it is specifically configured to: obtain a first similarity between the first query and the candidate knowledge document and a second similarity between the first query and the key document; and obtain the retrieval difficulty value of the key document retrieved according to the first query based on the difference between the first similarity and the second similarity.

[0108] Optionally, the processor 402 is further configured to: obtain from the key document the key sentence on which the first reference response result depends; and obtain the difficulty value of the first query test case based on the retrieval difficulty value, including: obtaining the integration difficulty value of integrating the key sentence to obtain the first reference response result based on the key sentence and the key document; and obtaining the difficulty value of the first query test case based on the retrieval difficulty value and the integration difficulty value.

[0109] Optionally, when the processor 402 obtains the difficulty value of the first query test case based on the key document, it is specifically configured to: obtain the key sentence from the key document on which the generation of the first reference response result depends; obtain the integration difficulty value of integrating the key sentence to obtain the first reference response result based on the key sentence and the key document; and obtain the difficulty value of the first query test case based on the integration difficulty value.

[0110] Optionally, when the processor 402 obtains the integration difficulty value for integrating the key sentence to obtain the first reference response result based on the key sentence and the key document, it is specifically configured to: obtain the information entropy corresponding to the key sentence based on the probability distribution of the key sentence in the key document; and obtain the integration difficulty value for integrating the key sentence to obtain the first reference response result based on the information entropy corresponding to the key sentence.

[0111] Optionally, when the processor 402 transforms the first query test case according to the entity associated with the first query test case to obtain the second query test case, it is specifically configured to: obtain the entities contained in the candidate knowledge document; obtain the associated document of the candidate knowledge document according to the entities contained in the candidate knowledge document and the association relationship between multiple knowledge documents in the knowledge base, wherein the association relationship between the multiple knowledge documents is determined according to the entities contained in each of the multiple knowledge documents; obtain the third query test case associated with the first query test case according to the associated document, wherein the third query test case contains a third question-answer pair, and the third question-answer pair includes: the third query and its corresponding third reference answer result; and transform the first query test case according to the third query test case to obtain the second query test case.

[0112] Optionally, when the processor 402 transforms the first query test case according to the third query test case to obtain the second query test case, it is specifically configured to: transform the first query according to the distribution characteristics of entities in the first question-and-answer pair and the third question-and-answer pair to obtain the transformed second query; call the content generation model to generate the second query and its corresponding transformed second reference response result; obtain the entities contained in the second query and the entities contained in the second reference response result to obtain the entities associated with the transformed second question-and-answer pair; filter out the transformed candidate knowledge documents in the knowledge base according to the entities associated with the second question-and-answer pair; and use the multivariate data group composed of the second question-and-answer pair and the transformed candidate knowledge documents as the second query test case.

[0113] Optionally, when the processor 402 transforms the first query based on the distribution characteristics of entities in the first question-and-answer pair and the third question-and-answer pair to obtain a transformed query, it is specifically configured to: if an entity in the first query is the third reference response result, then insert the first query into the third query to obtain the transformed query; or, if the semantic relationship between an entity in the first reference response result and the entity corresponding to the third reference response result satisfies a set condition, then integrate the first query and the third query to obtain the transformed query; or, if an entity in the first query is located in the third query, then concatenate the first query and the third query to obtain the transformed query; or, if an entity in the first reference response result is located in the third reference response result, then generate a query with the first reference response result and the third reference response result as the response result, as the transformed query.

[0114] Optionally, the second query test case includes: a second query, a second reference response result, and a transformed candidate knowledge document; the processor 402 is further configured to: call a content generation model to obtain key information required to respond to the second query, and obtain a transformed key document required to generate the second reference response result from the transformed candidate knowledge document; when the processor 402 evaluates the retrieval-enhanced model under test based on the second query test case and obtains the evaluation result, it is specifically configured to: input the second query and the transformed candidate knowledge document into the model under test, and obtain a target knowledge document obtained by the model under test from the transformed candidate knowledge document based on the second query, and a target response result obtained from responding to the second query based on the target knowledge document; determine the knowledge retrieval capability score of the model under test based on the difference between the transformed key document and the target knowledge document; and determine the content generation capability score of the model under test based on the matching degree between the target response result and the key information; and obtain the evaluation result based on the knowledge retrieval capability score and the content generation capability score.

[0115] In addition to the aforementioned embodiments, the electronic device shown in FIG4 can also be used to execute a method for generating test cases for a retrieval enhancement model. The processor 402 is specifically configured to: obtain a first query test case, the first query test case including: a first question-answer pair and its associated candidate knowledge document, the candidate knowledge document being determined based on the entity associated with the first question-answer pair; obtain the difficulty value of the first query test case based on the candidate knowledge document; if the difficulty value of the first query test case is less than or equal to a set difficulty value threshold, then transform the first query test case based on the entity associated with the first query test case to obtain a second query test case, the second query test case being used to evaluate the test model based on retrieval enhancement.

[0116] Furthermore, as shown in Figure 4, the electronic device also includes other components such as a communication component 403, a power supply component 404, a display component 405, and an audio component 406. Figure 4 only schematically shows some components and does not imply that the electronic device only includes the components shown in Figure 4. In Figure 4, the components within the dashed boxes are optional, not mandatory, and their specific inclusion depends on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or as a server-side device such as a conventional server, cloud server, or server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include the components within the dashed boxes in Figure 4; if the electronic device of this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, it may not include the components within the dashed boxes in Figure 4.

[0117] The memory 401 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0118] The communication component 403 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as Wi-Fi, 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0119] The power supply component 404 is configured to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0120] The display component includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0121] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0122] In this embodiment, the obtained query test cases include question-answer pairs and their associated candidate knowledge documents. Based on the candidate knowledge documents, the difficulty value of the query test cases can be obtained. If the difficulty value of a query test case is less than or equal to a set difficulty threshold, the query test case can be transformed based on the entities associated with it. The transformed query test case is then used to evaluate the retrieval-enhanced model under test, yielding evaluation results. The candidate knowledge documents are determined based on the entities associated with the question-answer pairs. Entities are a relatively universal document attribute, facilitating the mining of candidate knowledge documents related to the question-answer pairs based on knowledge documents with different structures and patterns. This improves the versatility of the evaluation method, allowing it to be flexibly transferred to different corpora. Furthermore, transforming the query test cases based on the entities associated with them enables dynamic adjustment of the evaluation difficulty value, ensuring that the evaluation results more accurately reflect the capabilities of the model under test.

[0123] Accordingly, this disclosure also provides a computer-readable storage medium storing a computer program, which, when executed, can perform the steps that can be executed by an electronic device in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0124] This disclosure also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the methods provided in this disclosure. It should be understood that each or a combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes said element.

[0126] The above description is merely an embodiment of this disclosure and is not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.

Claims

1. A model evaluation method, comprising: Obtain a first query test case, which includes: a first question-answer pair and its associated candidate knowledge documents, wherein the candidate knowledge documents are determined based on the entities associated with the first question-answer pair; Based on the candidate knowledge documents, obtain the difficulty value of the first query test case; If the difficulty value of the first query test case is less than or equal to the set difficulty value threshold, then the first query test case is transformed according to the entity associated with the first query test case to obtain the second query test case. Based on the second query test case, the model to be tested based on retrieval enhancement was evaluated, and the evaluation results were obtained.

2. The method according to claim 1, wherein, Obtain the first query test case, including: Retrieve at least one knowledge document from the knowledge base; Based on the at least one knowledge document, a content generation model is invoked to generate a first question-answer pair, the first question-answer pair including: a first query and its corresponding first reference answer result; Obtain the entities contained in the first query and the entities contained in the first reference response result to obtain the first entity associated with the first question and answer pair; Based on the first entity, the candidate knowledge documents are selected from the knowledge base; The first question-answer pair and the candidate knowledge documents are used as the multivariate data group for the first query test case.

3. The method according to claim 1, wherein, The first question-and-answer pair includes the first query and its corresponding first reference answer result; Based on the candidate knowledge documents, the difficulty value of the first query test case is obtained, including: From the candidate knowledge documents, obtain the key documents required to generate the first reference response result; Based on the key document and / or the candidate knowledge document, obtain the difficulty value of the first query test case.

4. The method according to claim 3, wherein, Based on the key document and the candidate knowledge document, obtain the difficulty value of the first query test case, including: The key documents are used as the expected search results; Based on the differences between the candidate knowledge documents and the key documents, obtain the retrieval difficulty value of the key documents retrieved according to the first query; Based on the search difficulty value, obtain the difficulty value of the first query test case.

5. The method according to claim 4, wherein, Based on the differences between the candidate knowledge documents and the key documents, the retrieval difficulty value of the key documents retrieved according to the first query is obtained, including: Obtain the first similarity between the first query and the candidate knowledge document, and the second similarity between the first query and the key document; Based on the difference between the first similarity and the second similarity, the retrieval difficulty value of the key document retrieved based on the first query is obtained.

6. The method according to claim 4, wherein, Also includes: From the key document, obtain the key sentences that the first reference response result depends on; Based on the retrieval difficulty value, the difficulty value of the first query test case is obtained, including: Based on the key sentences and the key documents, obtain the integration difficulty value of integrating the key sentences to obtain the first reference response result; Based on the retrieval difficulty value and the integration difficulty value, the difficulty value of the first query test case is obtained.

7. The method according to claim 3, wherein, Based on the key document, obtain the difficulty value of the first query test case, including: From the key document, obtain the key sentences that the first reference response result depends on; Based on the key sentences and the key documents, obtain the integration difficulty value of integrating the key sentences to obtain the first reference response result; Based on the integration difficulty value, obtain the difficulty value of the first query test case.

8. The method according to claim 6 or 7, wherein, Based on the key sentences and the key documents, obtain the integration difficulty value for integrating the key sentences to obtain the first reference response result, including: Based on the probability distribution of the key sentences in the key documents, obtain the information entropy corresponding to the key sentences; The integration difficulty value of the first reference response result is obtained by integrating the key sentences based on the information entropy corresponding to the key sentences.

9. The method according to any one of claims 2-7, wherein, Based on the entities associated with the first query test case, the first query test case is transformed to obtain the second query test case, including: Obtain the entities contained in the candidate knowledge document; Based on the entities contained in the candidate knowledge document and the relationships between multiple knowledge documents in the knowledge base, the associated documents of the candidate knowledge document are obtained. The relationships between the multiple knowledge documents are determined based on the entities contained in each of the multiple knowledge documents. Based on the associated document, obtain a third query test case associated with the first query test case. The third query test case contains a third question-answer pair, which includes the third query and its corresponding third reference response result. The first query test case is transformed based on the third query test case to obtain the second query test case.

10. The method according to claim 9, wherein, The first query test case is transformed based on the third query test case to obtain the second query test case, including: Based on the distribution characteristics of entities in the first question-and-answer pair and the third question-and-answer pair, the first query is transformed to obtain the transformed second query; The content generation model is invoked to generate the second query and its corresponding transformed second reference response result; Obtain the entities contained in the second query and the entities contained in the second reference response result to obtain the entities associated with the transformed second question-and-answer pair; Based on the entities associated with the second question-and-answer pair, the transformed candidate knowledge documents are selected from the knowledge base; The second question-and-answer pair and the transformed candidate knowledge documents are used as the multivariate data group for the second query test case.

11. The method according to claim 10, wherein, Based on the distribution characteristics of entities in the first and third question-answer pairs, the first query is transformed to obtain a transformed query, including: If the entity in the first query is the result of the third reference response, then the first query is inserted into the third query to obtain the transformed query; or, If the semantic relationship between the entity in the first reference response and the entity corresponding to the third reference response meets the set conditions, then the first query and the third query are integrated to obtain the transformed query; or, If the entity in the first query is also in the third query, then the first query and the third query are concatenated to obtain the transformed query; or, If the entity in the first reference response is located in the third reference response, then a query is generated with the first reference response and the third reference response as the response results, which is the transformed query.

12. The method according to any one of claims 3-7, wherein, The second query test case includes: a second query, a second reference response result, and a transformed candidate knowledge document; it also includes: calling a content generation model to obtain key information required to respond to the second query, and obtaining the transformed key document required to generate the second reference response result from the transformed candidate knowledge document; Based on the second query test case, the model to be tested based on retrieval enhancement was evaluated, and the evaluation results were obtained, including: The second query and the transformed candidate knowledge document are input into the model under test to obtain the target knowledge document obtained by the model under test from the transformed candidate knowledge document based on the second query, and the target response result obtained by responding to the second query based on the target knowledge document. Based on the differences between the transformed key documents and the target knowledge documents, the knowledge retrieval ability score of the model under test is determined; and, The content generation capability score of the model under test is determined based on the matching degree between the target response result and the key point information. The evaluation results are obtained based on the knowledge retrieval ability score and the content generation ability score.

13. A method for generating test cases for a retrieval enhancement model, comprising: Obtain a first query test case, which includes: a first question-answer pair and its associated candidate knowledge documents, wherein the candidate knowledge documents are determined based on the entities associated with the first question-answer pair; Based on the candidate knowledge documents, obtain the difficulty value of the first query test case; If the difficulty value of the first query test case is less than or equal to the set difficulty value threshold, then the first query test case is transformed according to the entity associated with the first query test case to obtain a second query test case. The second query test case is used to evaluate the model under test based on retrieval enhancement.

14. An electronic device, comprising: Memory and processor; The memory is configured to store one or more computer instructions; The processor is configured to execute one or more computer instructions to perform the following steps: Obtain a first query test case, which includes: a first question-answer pair and its associated candidate knowledge documents, wherein the candidate knowledge documents are determined based on the entities associated with the first question-answer pair; Based on the candidate knowledge documents, obtain the difficulty value of the first query test case; If the difficulty value of the first query test case is less than or equal to the set difficulty value threshold, then the first query test case is transformed according to the entity associated with the first query test case to obtain the second query test case. Based on the second query test case, the model to be tested based on retrieval enhancement was evaluated, and the evaluation results were obtained.

15. The electronic device according to claim 14, wherein, The processor is configured to execute the one or more computer instructions to: further perform the following steps: Retrieve at least one knowledge document from the knowledge base; Based on the at least one knowledge document, a content generation model is invoked to generate a first question-answer pair, the first question-answer pair including: a first query and its corresponding first reference answer result; Obtain the entities contained in the first query and the entities contained in the first reference response result to obtain the first entity associated with the first question and answer pair; Based on the first entity, the candidate knowledge documents are selected from the knowledge base; The first question-answer pair and the candidate knowledge documents are used as the multivariate data group for the first query test case.

16. The electronic device according to claim 15, wherein, The processor is configured to execute the one or more computer instructions to: further perform the following steps: Based on the candidate knowledge documents, the difficulty value of the first query test case is obtained, including: From the candidate knowledge documents, obtain the key documents required to generate the first reference response result; Based on the key document and / or the candidate knowledge document, obtain the difficulty value of the first query test case.

17. The electronic device according to claim 16, wherein, The processor is configured to execute the one or more computer instructions to: further perform the following steps: The key documents are used as the expected search results; Based on the differences between the candidate knowledge documents and the key documents, obtain the retrieval difficulty value of the key documents retrieved according to the first query; Based on the search difficulty value, obtain the difficulty value of the first query test case.

18. The electronic device according to claim 17, wherein, The processor is configured to execute the one or more computer instructions to: further perform the following steps: Obtain the first similarity between the first query and the candidate knowledge document, and the second similarity between the first query and the key document; Based on the difference between the first similarity and the second similarity, the retrieval difficulty value of the key document retrieved based on the first query is obtained.

19. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the implementation of the steps of the method according to any one of claims 1-13.

Images