Veracity Detection in Machine Comprehension via Corpus Alteration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine comprehension systems often produce nonsensical answers when the correct answer to a query is not present in the text corpus, leading to high confidence scores and reduced reliability.
Innovation Solution
A system that alters the text corpus to generate alternative answers and compares these with the initial answer to determine a veracity score, outputting a canned response if the score is below a threshold, indicating that the answer is not present.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the machine comprehension system generates an answer based on the text corpus, then the system provides a response to the query, but the answer may be nonsensical when the correct answer is not present in the corpus
Solution Approach 1:
The system performs preliminary actions by altering the text corpus in multiple ways before generating the final answer. It creates altered versions of the corpus (with modifications such as removing paragraphs, changing wording) and generates answers from these altered versions to compare with the original answer, thereby preliminarily assessing the reliability of the generated answer.
Solution Approach 2:
The system implements feedback by comparing the original answer with answers generated from altered corpora. If the answers differ significantly across multiple alterations, this feedback indicates the original answer may be nonsensical or hallucinated, triggering a reassessment of answer reliability and potentially returning a 'not present' response.
2Productivity
If the system always produces an answer regardless of presence in corpus, then the system maintains high productivity, but the confidence score becomes unreliable
Solution Approach 1:
Before finalizing the answer and confidence score, the system performs preliminary actions by generating multiple alternative answers from altered corpora. This preliminary comparison process allows the system to assess answer reliability before committing to a final response, thereby improving confidence score accuracy without significantly reducing productivity.
Solution Approach 2:
The system changes parameters of the text corpus (removing sections, altering wording, modifying structure) to generate varied versions. By observing how answer consistency changes across these parameter variations, the system can more accurately determine whether the original answer is truly supported by the corpus, thereby improving confidence score precision.
3Reliability
If the system generates alternative answers from altered corpora and compares them, then the veracity of the original answer is improved, but the device complexity increases
Solution Approach 1:
The system segments the verification process into distinct components: corpus alteration module, answer generation module, comparison module, and veracity assessment module. Each component performs a specific function, making the overall complex system more manageable and maintainable while achieving high answer veracity through coordinated operation of these segmented parts.
Data Source
AI summary
One or more systems, devices, computer program products and/or computer-implemented methods provided herein relate to determining veracity of answers generated by machine comprehension question and answer models. According to an embodiment, a machine comprehension component can generate a first answer to a query by extracting the first answer from a passage of text corpus. The text corpus alteration component can alter the text corpus one or more times to produce one or more altered text corpora. The machine comprehension component can further extract one or more additional answers to the query from the altered text corpora. A comparison component can determine a veracity score for the first answer based on one or more comparisons of the first answer with the one or more additional answers.