Query Variation Generation for RAG Evaluation Without Labeled Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limitations of prompt size in generative AI models like GPT3.5-Turbo, GPT4, and GPT-4-32k, and the lack of labeled data for training and evaluating Retrieval Augmented Generation (RAG) systems, result in inefficient and costly evaluation metrics, relying on manual effort and user feedback.
Innovation Solution
A genetic algorithm-driven query variation generation process that integrates user feedback to create diverse and relevant query-answer pairs, optimizing RAG systems by generating query variations and refining their performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If common indexing and retrieval techniques are used to match user queries to content portions, then the system can handle large document sizes, but the system lacks labeled data for training and evaluation
Solution Approach 1:
The system generates its own labeled training data by using the RAG system to retrieve content portions and automatically creating query-answer pairs. This self-service approach eliminates the need for external manual labeling while building a training dataset from the system's own operational data.
Solution Approach 2:
The system implements a feedback loop where user interactions and retrieval results are captured and used to generate labeled examples. This feedback mechanism continuously improves the system by converting operational data into training data, enhancing evaluation capabilities without external intervention.
2Reliability
If manual labeling is performed to create training data, then labeled data for evaluation is obtained, but significant financial cost and processing time are required
Solution Approach 1:
The system performs self-labeling by automatically generating query-answer pairs from its own retrieval operations. This eliminates the need for human annotators, reducing both financial cost and processing time while maintaining data quality through the system's inherent understanding of its domain.
Solution Approach 2:
The system creates copies of existing query-answer interactions and transforms them into labeled training examples. By replicating and adapting real user interactions, the system generates abundant training data without requiring new manual labeling efforts.
3Device complexity
If prompt size limits are applied to generative AI models, then the models can maintain manageable input sizes, but documents must be broken into smaller content portions
Solution Approach 1:
The system replaces mechanical text segmentation with semantic chunking based on embeddings and similarity metrics. This substitution preserves contextual relationships by grouping content portions based on semantic meaning rather than arbitrary size divisions, reducing information loss while maintaining manageable prompt sizes.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method, computer program product, and computing system for processing a plurality of query-answer pairs associated with a generative artificial intelligence (AI) model. A first set of query variations are generated from the plurality of query-answer pairs using a genetic algorithm. A plurality of content portions associated with the first set of query variations are identified using a Retrieval Augmentation Generation (RAG) system. A fitness score associated with each of the query variations of the first set of query variations is determined using the plurality of content portions. A plurality of query variation-answer pairs are generated by generating a second set of query variations from the first set of query variations using the genetic algorithm and the fitness scores associated with each of the first set of query variations.