Generative Model Output Error Detection via Ground Truth Comparison

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Natural language generation (NLG) systems, particularly those powered by large language models, often generate false information, including factual errors and hallucinations, which can be presented in a convincing manner, posing business and security risks due to the lack of effective automated detection methods.

Innovation Solution

A system and method for detecting errors and hallucinations in generative model output data by comparing it to ground truth information, using processors to generate data structures and indicate whether the output contains errors or hallucinations, allowing for automatic detection without requiring visibility into the generation process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If NLG systems generate output data, then productivity and creative power are improved, but the reliability of the output deteriorates due to factual errors and hallucinations

Engineering Contradiction:

Improveoutput generation capabilityVSAvoidaccuracy of output information

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements feedback by comparing generated output against ground truth data and providing error/hallucination detection results back to the user or system, enabling iterative improvement and trust building in NLG outputs

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

An intermediary verification system is introduced between the NLG model and the end user, which acts as a mediator to check factual accuracy and filter out hallucinations before the information reaches the user

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If manual human review is used for error detection, then the reliability of output is improved, but productivity and scalability deteriorate

Engineering Contradiction:

Improveerror detection accuracyVSAvoidreview speed and scalability

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent replaces the mechanical process of manual human review with an automated computational system that uses data structures and comparison algorithms to detect errors and hallucinations, eliminating the need for human intervention while maintaining high reliability

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Ease of operation

If NLG systems present false information confidently, then the persuasiveness and usefulness of output is improved, but the harmful effects of misinformation increase

Engineering Contradiction:

Improveconvincing nature of outputVSAvoidfalse information and hallucinations

Core Design Contradiction:

Ease of operationVSObject-generated harmful factors

Solution Approach 1:

The system changes the 'color' or visibility of false information by detecting and highlighting hallucinations and errors in the output, making them stand out from legitimate content so users can distinguish between accurate and fabricated information

Inventive Principle:

Principle #32Color changes

Data Source

PatentUS20240386207A1Systems and methods for detecting errors and hallucinations in generative model output data

Publication Date: 2024.11.21 NOBLIS INC

AI summary

Provided herein are systems and methods that can detect errors and hallucinations in output data produced by generative models. The detection systems described herein may compare information from the generative model output data with ground truth information to determine whether the generative model output data comprises errors and/or hallucinations. The systems and methods described herein may generate an output indicative of whether the generative model output data comprises errors and/or hallucinations. The systems and methods described herein can readily detect false information generated by NLG systems, thus potential harms of reliance on false information can be minimized.