RAG Hallucination Detection With Two-Stage Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods fail to accurately detect hallucinations in machine-generated responses, particularly in retrieval-augmented generation systems, leading to inefficiencies and inaccuracies in identifying misinformation.
Innovation Solution
Implementing a multilabel or two-tiered hallucination-detection system using models like DeBERTa and LLMs to classify machine-generated responses based on their consistency with a knowledge base, flagging unsupported statements for review.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing hallucination-detection methods are used, then the system can identify some misinformation, but the detection accuracy is insufficient and cannot reliably distinguish between supported and unsupported statements
Solution Approach 1:
The patent segments the hallucination detection task into two distinct classification stages: (1) determining whether a statement is verifiable or non-verifiable, and (2) for verifiable statements, determining whether it is supported or unsupported by the knowledge base. This segmentation allows each model to specialize in specific aspects of the detection task, improving overall accuracy and reliability compared to single-model approaches.
Solution Approach 2:
The patent introduces an intermediary verification step where the system first assesses whether a statement can be verified against the knowledge base before proceeding to support/unsupported classification. This intermediary layer acts as a filter that prevents misclassification of non-verifiable statements (such as opinions or subjective claims) as hallucinations, thereby improving detection reliability.
2Measurement precision
If a single hallucination-detection model is used, then the system structure remains simple, but the system cannot accurately handle different types of statements (verifiable vs. non-verifiable)
Solution Approach 1:
The detection system is segmented into two specialized classification models: a first model for verifiable vs. non-verifiable classification, and a second model for supported vs. unsupported classification. This segmentation enables each model to focus on specific statement types, achieving higher classification accuracy while maintaining manageable complexity through modular architecture.
Solution Approach 2:
Different parts of the detection system are assigned different functional qualities: the first classification model specializes in verifiability assessment, while the second model specializes in support verification. This local specialization allows each component to be optimized for its specific task, improving overall system precision without requiring a single overly complex model.
3Reliability
If manual review of all machine-generated responses is performed, then all potential hallucinations can be identified, but the agent response time increases significantly
Solution Approach 1:
The system implements self-service hallucination detection through automated multilabel classification models that independently assess machine-generated responses. The models automatically determine verifiability and support status without requiring manual review, thereby maintaining high reliability while preserving fast response times. Manual review is only needed for edge cases or disputed classifications.
Solution Approach 2:
The system provides automated feedback in the form of classification labels (verifiable/non-verifiable, supported/unsupported) that immediately indicate the reliability of generated responses. This feedback mechanism allows the system to self-correct or flag problematic outputs without human intervention, maintaining both reliability and speed in most cases.
Data Source
AI summary
Systems and methods may provide techniques for detecting hallucination in machine-generated responses. A computer-implemented method can include accessing text data. The text data can include one or more machine-generated responses that are supplemented by outputs generated by a retrieval-augmentation generation (RAG) system. In some instances, the outputs are associated with the prompt associated with a user. The computer-implemented method can also include applying one or more hallucination-detection models to the text data to generate a set of classification labels. A classification label can indicate whether a corresponding machine-generated response of the one or more machine-generated responses contradicts at least part of the knowledge base accessed by the RAG system. The computer-implemented method can also include generating annotated text data that includes the one or more machine-generated responses annotated with corresponding classification labels of the set of classification labels. The computer-implemented method can also include outputting the annotated text data.