Personalized Retriever Benchmarking With AI-Tuned User Scenarios
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing retrievers fail to accurately reflect user intent in search results due to a lack of evaluation methods that consider user information, and current benchmarks like BIER are inadequate for assessing instruction-following functionality and user instance relevance.
Innovation Solution
A system and method for constructing a dataset that includes virtual user scenarios and prompts to tune targets, ensuring they align with user information, using AI models to enhance the evaluation of retrievers' output relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing retrievers focus only on user query to output search results, then the retrieval process is simple and fast, but the search results fail to reflect user intent and information
Solution Approach 1:
The patent introduces an AI model as an intermediary component between the retriever and the evaluation process. This AI model generates virtual user scenarios and instructions that mediate the evaluation of whether search results reflect user intent, without requiring complex modifications to the retriever itself
Solution Approach 2:
The patent creates virtual user scenarios that copy and simulate real user characteristics, intents, and preferences. These synthetic user profiles serve as proxies for actual users, enabling evaluation of retriever performance on user intent reflection without needing to collect and process real user data
2Adaptability or versatility
If BIER benchmark is used to evaluate retrievers by search task, then evaluation can be performed with existing data, but it cannot assess instruction-following functionality and user instance relevance
Solution Approach 1:
The patent performs preliminary actions by pre-generating comprehensive virtual user scenarios and instructions before the actual evaluation process. This preparation includes creating diverse user profiles with different characteristics, preferences, and search intents, which then serves as the foundation for scalable evaluation
Solution Approach 2:
The patent transitions from evaluating retrievers in a single dimension (search task accuracy) to multiple dimensions by incorporating user characteristics, preferences, and instruction-following capabilities. This adds new evaluation dimensions that capture user information personalization and intent reflection
3Measurement precision
If the number of instances used for evaluation is small, then the evaluation process is quick and simple, but the evaluation is not appropriate for assessing instruction-following functionality
Solution Approach 1:
The patent implements self-service by using the AI model to automatically generate virtual user scenarios, instructions, and evaluation data without requiring manual creation. The system serves itself by autonomously producing the evaluation dataset, reducing human time investment while maintaining precision
Data Source
AI summary
A system, method, and program for constructing a dataset to evaluate user information personalization functionality of retrievers. The method includes extracting a plurality of queries and a target corresponding to each of the plurality of queries from sample data, inputting a first prompt into an Artificial Intelligence (AI) model to output an instruction set composed of a plurality of instructions including virtual user scenarios, additionally associating the instruction set with each of the corresponding plurality of queries and target to output as element data, inputting the element data together with a second prompt into the AI model to tune the target included in the element data to fit the virtual user scenario included in the plurality of instructions, and storing the plurality of tuned element data as a dataset.