Answer Key Verification in Cognitive Question Answering Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The existing solutions for generating and verifying accurate question and answer pairs for cognitive question answering systems are inefficient and prone to errors due to manual fact-checking, leading to inaccuracies in training data that impair the system's accuracy.

Innovation Solution

A system and method that uses a similarity calculation engine to compare answer key answers with returned answers from a QA system, identifying potential problems by computing similarity metric values and updating the answer key with gradient characteristics from the returned answers, thereby enhancing the accuracy of training data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual fact checking is used to verify training QA pairs, then answer key accuracy can be improved, but the time and resources required increase significantly

Engineering Contradiction:

Improveanswer key accuracyVSAvoidfact checking time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The QA system automatically verifies answer key accuracy by comparing candidate answers against the answer key without requiring external manual fact-checking. The system uses its own cognitive capabilities to perform the verification that would otherwise need to be done manually by human experts.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system implements a feedback mechanism where the QA system's returned answers are compared to the answer key, and discrepancies are used to identify and correct answer key problems. This automated feedback loop continuously improves answer key accuracy without additional manual intervention.

Inventive Principle:

Principle #23Feedback

2Reliability

If manual answer correction processing is performed to correct inaccuracies in training QA pairs, then answer key accuracy can be improved, but the process becomes cumbersome and error-prone

Engineering Contradiction:

Improveanswer key accuracyVSAvoidcorrection process ease

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The system automatically identifies and flags answer key problems by comparing returned answers with the answer key, eliminating the need for manual correction processes. The cognitive system performs self-verification and highlights discrepancies for automated processing rather than requiring manual intervention.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical correction processes with automated computational comparison. Instead of human operators manually reviewing and correcting answer keys, the system uses algorithmic similarity computation and automated flagging to identify and correct inaccuracies.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If comprehensive fact checking is performed on all training QA pairs, then answer key accuracy can be improved, but the resources required increase significantly

Engineering Contradiction:

Improveanswer key accuracyVSAvoidtraining data generation efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The QA system performs self-verification of its own training data by comparing returned answers against the answer key. This automated self-checking mechanism maintains high answer key accuracy without requiring external fact-checking resources, thereby preserving training data generation efficiency.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Instead of performing comprehensive fact-checking on all training QA pairs, the system applies targeted verification only where needed - comparing returned answers against the answer key to identify specific discrepancies. This partial verification approach maintains accuracy while preserving overall productivity.

Inventive Principle:

Principle #16Partial or excessive action

4Productivity

If automated comparison of answer keys is implemented, then verification efficiency can be improved, but the system complexity increases

Engineering Contradiction:

Improveverification efficiencyVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The QA system uses its existing multi-functional cognitive capabilities to perform both question answering and answer key verification. The same natural language processing and reasoning mechanisms used for generating answers are also employed for comparing and verifying answer keys, avoiding the need for separate verification infrastructure.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10795878B2System and method for identifying answer key problems in a natural language question and answering system

Publication Date: 2020.10.06 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A system and a computer program product are provided for evaluating question-answer pairs in an answer key by comparing a first answer key answer to a plurality of candidate answers to determine if the answer key may have a problem if the plurality of candidate answers are more similar to one another than to the first answer and to determine if the plurality of candidate answers has gradient information which may be used to update the answer key if not already included in the answer key.