Retriever Evaluation Dataset Using Virtual User Scenarios
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems lack an effective means to evaluate whether information retrievers output search results that reflect user intent and preferences, as current benchmarks like BEIR do not adequately assess instruction-following functionality and are limited by small instance numbers.
Innovation Solution
A system and method using an AI model to construct a dataset by extracting queries and targets from sample data, associating them with virtual user scenarios, and tuning targets to fit these scenarios, with prompts to refine and score the data for consistency and relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If BEIR benchmark is used to evaluate retrievers, then evaluation can be performed by search task, but it cannot adequately assess instruction-following functionality and user intent reflection
Solution Approach 1:
The patent segments the evaluation process into multiple components: extracting queries and targets from sample data, generating virtual user scenarios with specific instructions, associating instruction sets with query-target pairs, and tuning targets to fit user scenarios. This segmentation allows each component to be optimized independently while collectively achieving comprehensive user intent evaluation.
Solution Approach 2:
The patent introduces virtual user scenarios as an intermediary between the retriever system and the evaluation process. These scenarios act as mediators that translate abstract user intents into concrete, evaluable instances with specific instructions, queries, and targets, enabling precise measurement of instruction-following functionality.
2Measurement precision
If BEIR benchmark uses limited number of instances, then evaluation process is simple, but evaluation results lack statistical significance and generalizability
Solution Approach 1:
The patent performs preliminary actions by pre-extracting queries and targets from sample data, pre-generating virtual user scenarios with comprehensive instructions, and pre-associating instruction sets with query-target pairs before the actual evaluation. This preliminary preparation enables the use of large-scale instances without overwhelming the evaluation process complexity.
Solution Approach 2:
The patent creates multiple copies of evaluation instances by generating numerous virtual user scenarios from a limited set of sample data. Each scenario is a copied and adapted version that maintains the structural integrity of the original while introducing variations in user intent, instructions, and contexts, thereby achieving statistical significance through replication.
3Manufacturing precision
If retrievers focus only on user query without user information, then retrieval process is simple and fast, but search results cannot reflect user intent and preferences
Solution Approach 1:
The patent applies local quality by integrating user information processing specifically at the evaluation stage rather than throughout the entire retrieval system. The virtual user scenarios and instruction sets provide localized enhancement to specific query-target pairs, allowing precise measurement of user intent reflection without requiring complex modifications to the overall retriever architecture.
Solution Approach 2:
The patent adds another dimension to the evaluation by introducing virtual user scenarios that incorporate user background, location, occupation, hobby, interest, search goal, and preferred source. This dimensional expansion transforms the evaluation from simple query-result matching to comprehensive user intent assessment, enabling precise measurement of instruction-following functionality.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system, method, and program for constructing a dataset to evaluate user information personalization functionality of retrievers are disclosed. The method includes extracting a plurality of queries and a target corresponding to each of the plurality of queries from sample data, inputting a first prompt into an AI model to output an instruction set composed of a plurality of instructions including virtual user scenarios, additionally associating the instruction set with each of the corresponding plurality of queries and target to output as element data, inputting the element data together with a second prompt into the AI model to tune the target included in the element data to fit the virtual user scenario included in the plurality of instructions, and storing the plurality of tuned element data as a dataset.