LLM Hallucination Detection via Verification Data Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large Language Models (LLMs) can 'hallucinate' by providing incorrect information or inventing data in response to prompts, which can lead to inaccurate results in tasks such as cybersecurity data analysis.

Innovation Solution

Incorporating a verification request in the prompt for the LLM, which instructs the model to generate verification data showing how the query result is derived from the input data, and processing this verification data to validate the query result.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If LLMs are used to analyze cybersecurity data, then the model can provide rapid responses and summaries, but the model may hallucinate data not present in the input, leading to incorrect information

Engineering Contradiction:

Improveresponse speedVSAvoiddata accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements a verification mechanism where the LLM generates verification data along with its response, and this verification data is then checked against the original input data to confirm accuracy. This feedback loop allows the system to maintain high productivity while detecting and preventing hallucinations.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary verification by asking the LLM to generate verification data before finalizing the response. This preliminary action ensures that the information can be validated against the input data, preventing hallucinations from compromising reliability.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If a verification request is added to the prompt, then the accuracy of query results can be improved, but the complexity of the prompt processing increases

Engineering Contradiction:

Improvequery result accuracyVSAvoidprompt processing complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The verification process is segmented into distinct components: the original query, the verification request, the LLM's response containing both query results and verification data, and the final validation step. This segmentation makes the complex verification process more manageable and systematic.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The verification data acts as an intermediary element between the input data and the final query result. It serves as a bridge that allows systematic verification without requiring direct complex processing of the entire prompt, thereby managing complexity while improving accuracy.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250068667A1Identifying hallucinations in large language model output

Publication Date: 2025.02.27 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A computer-implemented method of generating verification data for a query result provided by a large language model, LLM, includes generating a prompt for the large language model. The prompt contains a verification request for a query, the query including query text and input data from which the query result can be derived. The verification request includes instructions that cause the LLM to generate verification data that indicates a derivation of the query result from the input data. Another computer-implemented method includes receiving the verification data and processing the verification data to determine whether the query result was validly derived from the input data.