Concept Expansion System Accuracy Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evaluating the accuracy of concept expansion systems is challenging due to the lack of compatible test datasets, which is a manual, labor-intensive and cost-inefficient process.
Innovation Solution
Automating the creation of test datasets by transforming a corpus of text from a specific domain into a test dataset and expected dataset, using annotators to map annotated text to compatible concepts, and calculating accuracy metrics like precision and recall based on the intersection between expected and returned concepts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual labeling is used to create test datasets for evaluating concept expansion systems, then evaluation accuracy can be ensured, but the process becomes labor-intensive and cost-inefficient
Solution Approach 1:
The system uses the concept expansion system itself to generate expected output concepts by processing annotated text, eliminating the need for manual expert labeling. The system serves itself by using its own processing capabilities to create evaluation data
Solution Approach 2:
The system pre-processes text corpora with annotators to create annotated text and expected concept sets before actual evaluation, preparing all necessary evaluation materials in advance through automated processing rather than manual creation
2Reliability
If manual creation of test datasets is performed to ensure data compatibility with concept expansion systems, then data quality is improved, but time consumption and costs increase significantly
Solution Approach 1:
The patent replaces manual mechanical processes of expert labeling and dataset curation with automated computational processes including annotators, text processing algorithms, and automated concept extraction systems
Solution Approach 2:
The system creates copies of annotated text and concepts through automated processing, generating multiple test datasets from single text corpora without requiring manual recreation for each evaluation scenario
Data Source
AI summary
Accuracy evaluation of concept expansion systems is provided. A returned set of related concepts corresponding to a test set of concepts is received from a concept expansion system. A comparison is performed between an expected set of concepts and the returned set of related concepts corresponding to the test set of concepts to identify intersection. An accuracy metric corresponding to the concept expansion system is generated based on the intersection between the expected set of concepts and the returned set of related concepts corresponding to the test set of concepts. It is determined whether the accuracy metric is greater than a minimum accuracy threshold level. A set of action steps is performed in response to determining that the accuracy metric is not greater than the minimum accuracy threshold level.