Ground Truth Ranking for Question-Answering Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developing a ground truth for question-answering systems is time-consuming and challenging due to the need for human annotators to identify appropriate answers from thousands of documents, where answers can be indirect, contain extraneous material, or be less than perfect, with current systems focusing mainly on perfect answers and lacking in collecting and ranking acceptable ones.
Innovation Solution
A computer-implemented method and system that allows annotators to use a document search and discovery tool to define patterns for answer passages, retrieve relevant documents, and rank answers on a scale from acceptable to perfect, using a domain dictionary with lexicons, glossaries, and knowledge base queries, enabling the identification of both perfect and acceptable answers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If human annotators manually identify all appropriate answers from thousands of documents, then the ground truth quality is improved, but the time consumption and workload increase significantly
Solution Approach 1:
The system performs preliminary automated retrieval of candidate answer passages from the document corpus before human annotators review them. This pre-processing step filters and organizes potential answers, so annotators only need to evaluate a manageable subset rather than searching through all thousands of documents manually, thus maintaining quality while reducing time consumption
Solution Approach 2:
The patent introduces an intermediary automated retrieval system that acts as a bridge between the large document corpus and human annotators. This intermediary component pre-processes the data by identifying and presenting relevant candidate passages, reducing the burden on annotators while ensuring they receive comprehensive candidate sets for evaluation
2Ease of manufacture
If the system focuses only on perfect answers, then the annotation process is simplified, but the ground truth becomes incomplete by excluding acceptable but imperfect answers
Solution Approach 1:
The patent changes the parameter of answer evaluation from a binary perfect/imperfect classification to a multi-level ranking system with at least two levels: perfect answers and acceptable answers. This parameter change allows the system to capture both high-quality and partially relevant answers, making the ground truth more complete while providing annotators with clear guidance through the ranking framework
Solution Approach 2:
The evaluation process is segmented into distinct ranking levels (perfect and acceptable answers). This segmentation allows annotators to systematically categorize answers based on their quality, making the annotation process more structured and manageable while ensuring both perfect and acceptable answers are captured in the ground truth
3Productivity
If annotators must immediately judge and rank all candidate answers, then the ground truth is finalized quickly, but the accuracy of ranking decreases due to annotator fatigue and cognitive load
Solution Approach 1:
The system performs preliminary retrieval and organization of candidate answers before the annotation phase. By pre-processing and presenting answers in an organized manner, annotators start with a manageable and well-structured set of candidates, reducing cognitive load and enabling more accurate ranking decisions while maintaining efficient workflow
Solution Approach 2:
The patent implements a feedback mechanism where annotators review candidate answers and provide rankings, which then inform further refinement of the ground truth. This iterative feedback process allows annotators to adjust their evaluations based on patterns observed in previous answers, improving ranking accuracy over time while maintaining productivity through the structured feedback loop
Data Source
AI summary
A computer-implemented method for ranking answers of question-answer pairs in ground truth. The computer-implemented method includes annotating, via an annotation tool, the domain dictionary, including, at least one or more of lexicons, glossaries, knowledge base query for ranking the question-answer pairs. The computer-implemented method further includes defining search patterns of the annotated domain dictionary. The computer-implemented method further includes searching a documents corpus or knowledge based query of the domain dictionary for candidate answers of the question-answer pairs, based on the defined search patterns, the candidate are returned based on the search. The computer-implemented method for evaluating the candidate answers with the annotation tool to verify correctness or acceptability of the returned candidate answers based on the search.