LLM Hallucination Detection Using Entity Pair Semantic Comparison

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Generative language models often produce factually inaccurate or misleading responses, known as hallucinations, which are unacceptable in domains like aviation where accuracy is critical, and current methods to combat hallucinations require unsustainable human intervention.

Innovation Solution

A computing system that uses entity extraction, semantic pairing, and semantic similarity models to compare entities in generative AI output with source text, identifying hallucinations by forming semantic pairs and comparing them against a domain-specific knowledge base.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If generative language models are used to produce text output, then productivity is improved, but reliability deteriorates due to hallucinations

Engineering Contradiction:

Improvetext generation efficiencyVSAvoidfactual accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent introduces an intermediary verification system that includes an entity extraction model, semantic pairing model, and semantic similarity model. These components act as mediators between the generative LLM and the final output, extracting entities from both the generated text and source text, forming semantic pairs, and comparing them to detect hallucinations before the text is finalized.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If human intervention is increased to combat hallucinations, then reliability is improved, but device complexity and loss of time worsen

Engineering Contradiction:

Improveinformation accuracyVSAvoidhuman intervention requirements

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system implements self-service by enabling automatic hallucination detection through machine learning models. The entity extraction model, semantic pairing model, and semantic similarity model work autonomously to identify potential hallucinations without requiring human reviewers, thus maintaining reliability while reducing human intervention complexity.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent incorporates feedback mechanisms where the semantic similarity comparison results are used to identify hallucinations, which can then be fed back to improve the generative model. The system provides feedback loops that allow continuous improvement of accuracy while reducing the need for manual intervention over time.

Inventive Principle:

Principle #23Feedback

3Reliability

If human review is used to verify AI output, then reliability is improved, but loss of time worsens

Engineering Contradiction:

Improveresponse accuracyVSAvoidverification time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent replaces the mechanical human review process with automated computational models. The entity extraction model, semantic pairing model, and semantic similarity model perform verification tasks that would otherwise require human time, dramatically reducing verification time while maintaining or improving reliability through systematic automated comparison.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Productivity

If generative models are used to reduce drafting load, then productivity is improved, but object-generated harmful factors worsen due to hallucination risks

Engineering Contradiction:

Improvedrafting efficiencyVSAvoidhallucination risk

Core Design Contradiction:

ProductivityVSObject-generated harmful factors

Solution Approach 1:

The patent applies preliminary anti-action by proactively detecting and flagging potential hallucinations before they become harmful output. The semantic similarity comparison is performed in advance on all generated entities, identifying discrepancies between source text entities and generated text entities, thereby preventing harmful hallucinated information from being disseminated.

Inventive Principle:

Principle #9Preliminary anti-action

Data Source

PatentUS20260017308A1Computing system for identifying hallucinations in generative artificial intelligence output

Publication Date: 2026.01.15 THE BOEING CO

AI summary

A computing system for identifying hallucinations in generative artificial intelligence (AI) output includes processing circuitry configured to receive a text output generated by a generative large language model (LLM) in response to an input prompt including origin source text data, and using an entity extraction model, extract entities from the text output and from the origin source text data. The processing circuitry, using a semantic pairing model, forms first semantic pairs from the entities of the text output and second semantic pairs from the entities of the origin source text data, and using a semantic similarity model, semantically compares the first semantic pairs with the second semantic pairs. The processing circuitry, based on at least the comparison, classifies whether or not any of the first semantic pairs is a hallucination, and outputs an indication of the classification.