Reasoning Graph Provenance Tracking for LLM Hallucination Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI systems, particularly Large Language Models (LLMs), are susceptible to hallucinations and lack effective methods for assessing confidence in their outputs, especially when dealing with unstructured text analysis, leading to inaccuracies and challenges in generating reliable reasoning graphs.
Innovation Solution
A networked computer system that generates structured reasoning graphs by decomposing questions into steps, calculating confidence levels deterministically, and tracking provenance to detect hallucinations and guide user intervention, using a combination of sampling-based and deterministic methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If LLMs are used for unstructured text analysis, then complex high-level reasoning tasks can be performed, but hallucinations occur and confidence assessment is unreliable
Solution Approach 1:
The patent segments the reasoning process into discrete steps, each with its own confidence score and provenance tracking. This allows systematic evaluation of each reasoning component rather than treating the entire output as a black box, thereby improving confidence assessment while maintaining reasoning capability.
Solution Approach 2:
The system incorporates feedback mechanisms that continuously monitor and evaluate reasoning steps, adjusting confidence scores based on provenance analysis. This feedback loop enables the system to identify and correct hallucinations while maintaining the ability to perform complex reasoning tasks.
2Reliability
If probabilistic classifiers are used for confidence assessment, then confidence scores can be generated, but the complexity of determining important tokens and handling multiple expressions increases
Solution Approach 1:
The patent extracts confidence assessment from the complex probabilistic classification process and implements it through deterministic provenance tracking. By separating the confidence calculation from the reasoning process, the system achieves reliable confidence scores without the complexity of token-level probabilistic analysis.
Solution Approach 2:
The system changes the parameter used for confidence assessment from probabilistic outputs to deterministic provenance tracking. This parameter transformation simplifies the calculation process while maintaining reliability, as provenance can be tracked through structured data rather than probabilistic distributions.
3Reliability
If black-box approaches like SelfCheckGPT are used, then hallucination detection can be performed, but access to intermediate calculations and reasoning steps is lost
Solution Approach 1:
The patent introduces provenance tracking as an intermediary mechanism that captures intermediate calculations and reasoning steps without requiring access to the model's internal state. This mediator allows hallucination detection while preserving information about the reasoning process, enabling both reliability and transparency.
Solution Approach 2:
The system performs preliminary action by recording provenance information at each reasoning step before the final output is generated. This preliminary tracking ensures that intermediate calculations are preserved and available for analysis, eliminating the need for post-hoc inspection of model internals.
4Ease of operation
If string or number matching techniques are used for answer matching, then trivial matches can be identified, but complex reasoning and evidence tracking become difficult
Solution Approach 1:
The patent adds a dimensional layer to answer matching by incorporating provenance tracking alongside traditional string or number matching. This additional dimension enables the system to handle complex reasoning and evidence tracking while maintaining the simplicity of basic matching techniques, as provenance provides structured context for interpretation.
Data Source
AI summary
A networked computer system for generating Reasoning Graphs is described herein. The networked computer system includes a data storage server storing a data source including information associated with a plurality of evidence documents and a data analysis computer server including one or more data analysis processors coupled to the data storage server and to an artificial intelligence (AI) computer system. The one or more data analysis processors programmed to execute an algorithm including the steps of querying the AI computer system to determine one or more entry-level answers based on the extracted evidence from the plurality of evidence documents and generating a reasoning graph data structure by determining a corresponding confidence score associated with each entry-level answer and identifying a corresponding evidence document used in determining each entry-level answer.