Ground Truth Evaluation via Automated Consistency Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Question and Answer (QA) systems face challenges in quickly assessing the quality of ground truth provided by subject matter experts due to long feedback cycles, which hinder the improvement of training data quality and subsequent response accuracy.
Innovation Solution
A method that involves providing training questions to both QA systems and subject matter experts, generating and comparing feature scores based on relevance, identifying consistent scoring features, and providing feedback to experts to enhance the quality of ground truth, thereby improving the ranking and relevance of answers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual inspection by experts is used to evaluate ground truth quality, then evaluation accuracy is improved, but feedback cycle time increases
Solution Approach 1:
The system implements automated feedback loops where the QA system evaluates ground truth quality and provides immediate feedback to experts, eliminating long manual review cycles while maintaining evaluation accuracy through multiple scoring features and consistency measurements
Solution Approach 2:
Manual expert inspection is replaced with an automated QA system that uses natural language processing, information retrieval, and machine learning to evaluate ground truth quality, significantly reducing feedback time while maintaining or improving evaluation precision
2Reliability
If extensive ground truth evaluation is performed to ensure quality, then answer accuracy is improved, but system complexity increases
Solution Approach 1:
The evaluation process is segmented into multiple independent scoring features (question type, focus, lexical answer type, sentence structure, grammatical modifiers) that can be computed separately and then aggregated, making the complex evaluation manageable and scalable
Solution Approach 2:
The QA system performs multiple functions simultaneously - it answers questions, evaluates ground truth quality, measures expert consistency, and provides feedback - reducing the need for separate specialized systems and lowering overall complexity
Data Source
AI summary
A method for training a question answering system includes providing training questions to a question answering system executing on a computer and to a plurality of subject matter experts. The question answering system generates first answers to each training question. Second answers to the training questions are received from the subject matter experts. Feature scores for each of the first answers and the second answers are generated and compared across the second answers and the first answers. Each of the feature scores is representative of a quality of an answer that is indicative of relevance to a corresponding training question. Based on the comparing, a measure of consistency of the feature scores of the second answers is determined, and a measure of consistency of the feature scores of the second answers to the first answers is determined. The measures of consistency are transmitted to the subject matter experts.