LLM Relevance Judging for RAG Retriever Feedback Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Retrieval Augmented Generation (RAG) bot systems lack a mechanism to refine the document retriever module based on user feedback, leading to inefficiencies and inaccuracies in document retrieval and response generation.
Innovation Solution
Implement an LLM-as-a-judge system to evaluate user feedback on chatbot responses, generating training data pairs to refine the RAG bot's document retrieval model by identifying relevant and irrelevant documents, thereby improving the model's accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If RAG bot systems use automated document retrieval without user feedback mechanisms, then the system operates with simplified architecture, but the retrieval accuracy and model refinement capability deteriorate
Solution Approach 1:
The patent implements a feedback mechanism where user interactions with chatbot responses are collected and used to train the document retriever model. Users can indicate whether retrieved documents were helpful or not, and this feedback is fed back into the system to continuously improve retrieval accuracy through model retraining.
Solution Approach 2:
The system automatically collects user feedback data and uses it to retrain the retriever model without requiring manual intervention. The chatbot itself generates training data by logging user interactions, and the system autonomously performs model refinement, reducing the need for external annotation services.
2Adaptability or versatility
If RAG bot systems collect and process user feedback for continuous improvement, then model refinement capability improves, but the facility for processing feedback and system complexity worsens
Solution Approach 1:
The system automatically logs user interactions and generates training data without manual intervention. The chatbot autonomously collects feedback data, processes it through the training pipeline, and retrains the model, eliminating the need for external annotation services and reducing operational complexity.
Solution Approach 2:
The patent introduces a training data generator component that acts as an intermediary between user feedback and model retraining. This component automatically processes user interactions, generates structured training data, and feeds it to the training pipeline, simplifying the overall feedback processing architecture.
3Measurement precision
If direct user feedback on retrieved documents is implemented, then relevance assessment accuracy improves, but user interaction complexity and system overhead worsen
Solution Approach 1:
The system implements a feedback mechanism where users can indicate whether retrieved documents were helpful or not. This feedback is collected and used to train the retriever model, improving relevance assessment accuracy while maintaining simple user interaction through straightforward feedback options.
4Manufacturing precision
If manual annotation services are used for training data generation, then training data quality improves, but cost and time consumption worsen
Solution Approach 1:
The system automatically generates training data by logging user interactions with the chatbot. User feedback on document relevance is automatically captured and structured into training datasets, eliminating the need for manual annotation services and significantly reducing both time and cost while maintaining data quality through real-world user perspectives.
Data Source
AI summary
There are provided systems and methods for identifying relevance of documents for automated retrieval models using large language models. An online transaction processor or other service provider may provide computing services and platforms to entities, which may include chatbots, information retrieval systems, question-and-answer systems, and the like. To provide better retrieval model training and refinement, the service provider may generate training data from user interaction logs, which may include user feedback that may be used to determine if documents are relevant to queries, and therefore should be retrieved for answering those queries by automated retrieval models. An LLM may be used as a judge to determine whether chatbot responses reference certain document. If not references, the query may be analyzed to determine whether certain retrieved documents are relevant. Data pairs may be generated for the training data from these processes and used for model refinement.