Multi-Hop Claim Verification Using HoVer Evidence Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing fact-checking systems struggle with multi-hop reasoning, as they often rely on single-source information and are prone to word-matching reasoning shortcuts, limiting their effectiveness in verifying claims that require evidence from multiple documents.
Innovation Solution
A machine learning system and dataset (HoVer) that challenges models to extract facts from multiple textual sources and classify claims as supported or not supported, using a 3-hop or 4-hop claim verification process with diverse reasoning graphs, enhancing accuracy through complex multi-hop reasoning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single-source information is used for fact-checking, then the system complexity is reduced, but the accuracy of claim verification deteriorates when multiple documents are required
Solution Approach 1:
The patent segments the fact-checking process into distinct reasoning hops, where each hop corresponds to retrieving and processing information from a specific document or source. This segmentation allows the system to handle multi-document verification by breaking down the complex task into manageable steps, each handled by specialized modules that process individual sources before integrating results.
Solution Approach 2:
The patent introduces a multi-hop reasoning dimension to traditional single-hop fact-checking systems. By transitioning from verifying claims based on one document to verifying across multiple documents through sequential reasoning hops, the system adds a temporal and structural dimension to the verification process, enabling accurate handling of claims that require synthesis across multiple sources.
2Speed
If word-matching reasoning shortcuts are used, then the processing speed is improved, but the accuracy deteriorates in adversarial evaluation
Solution Approach 1:
The patent incorporates feedback mechanisms that evaluate the quality of reasoning at each hop and provide corrections or adjustments. The system monitors whether retrieved documents and extracted facts align with the claim being verified, and uses this feedback to adjust subsequent retrieval and reasoning steps, preventing reliance on spurious word-matching patterns while maintaining processing efficiency.
Solution Approach 2:
The patent replaces simple mechanical word-matching operations with more sophisticated semantic reasoning mechanisms. Instead of relying solely on surface-level text matching, the system uses contextual understanding, entity resolution, and logical inference to verify claims, substituting the mechanical approach with a more intelligent process that maintains speed while improving accuracy against adversarial examples.
3Reliability
If multi-hop reasoning is implemented, then the accuracy of claim verification is improved, but the device complexity increases
Solution Approach 1:
The patent designs a universal fact-checking framework where the same core modules serve multiple functions across different reasoning hops. The document retrieval, sentence extraction, and claim verification components are designed to operate consistently across hops, reducing overall system complexity by avoiding the need for separate specialized systems for each reasoning step while still enabling accurate multi-hop verification.
4Ease of operation
If single-hop fact extraction is used, then the ease of operation is improved, but the adaptability to multi-source claims deteriorates
Solution Approach 1:
The patent implements a dynamic fact-checking system that automatically adjusts the number of reasoning hops based on the complexity of the claim and the documents required for verification. The system can operate in single-hop mode for simple claims to maintain ease of operation, while dynamically transitioning to multi-hop mode when the claim requires evidence from multiple sources, thus providing adaptability without sacrificing simplicity for routine cases.
Data Source
AI summary
Machine learning (ML) systems and methods for fact extraction and claim verification are provided. The system receives a claim and retrieves a document from a dataset. The document has a first relatedness score higher than a first threshold, which indicates that ML models of the system determine that the document is most likely to be relevant to the claim. The dataset includes supporting documents and claims including a first group of claims supported by facts from more than two supporting documents and a second group of claims not supported by the supporting documents. The system selects a set of sentences from the document. The set of sentences have second relatedness scores higher than a second threshold, which indicate that the ML models determine that the set of sentences are most likely to be relevant to the claim. The system determines whether the claim includes facts from the set of sentences.