Corpus Quality Analysis for NLP Augmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing question answering systems face challenges in determining the quality and relevance of new corpora for augmentation, as they lack methods to assess which parts of the corpus are most useful and whether new data sets are worth licensing, given the costs and legal efforts involved.
Innovation Solution
A mechanism for corpus quality analysis is introduced, which uses filters to evaluate the quality of new corpora by considering general readability, metadata, and domain/content analysis, determining which corpora to pursue and how much to pay based on key factors, leveraging machine learned features and characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If new corpora are added to augment existing corpora for QA systems, then the quantity and quality of training data increases, but the cost and legal effort for licensing increases
Solution Approach 1:
The system performs preliminary quality assessment of candidate corpora using multiple filters (readability filter, metadata filter, domain filter, content filter) before actually licensing and integrating them. This preliminary screening action identifies high-quality corpora that are likely to provide good QA results, allowing the system to invest licensing resources only in corpora that pass the quality threshold, thus resolving the contradiction between obtaining more training data and controlling licensing costs.
Solution Approach 2:
The system establishes a feedback loop where QA results from using a corpus are fed back into the quality assessment process. By monitoring which corpora produce better QA results, the system can refine its quality filters and prioritize licensing of corpora that demonstrably improve system performance, making the investment in licensing more effective and justifiable.
2Measurement precision
If comprehensive quality assessment filters are applied to evaluate corpus quality, then the accuracy of corpus selection improves, but the processing time and complexity increases
Solution Approach 1:
The comprehensive quality assessment is segmented into multiple independent filters that can be applied in sequence: readability filter, metadata filter, domain filter, and content filter. Each filter handles a specific aspect of quality assessment and can independently reject corpora that fail to meet minimum thresholds. This segmentation allows the system to maintain high assessment accuracy while reducing processing time by eliminating不合格 corpora at early stages rather than analyzing all corpora through all filters in full detail.
Solution Approach 2:
The system applies partial action by implementing filters with varying levels of stringency and computational cost. Critical filters (like readability and domain relevance) are applied thoroughly, while less critical filters (like certain metadata checks) are applied more lightly. This partial application of comprehensive assessment ensures sufficient accuracy for good corpus selection without the full processing time cost of equally thorough assessment of every possible corpus attribute.
3Productivity
If machine learned features are used to assess corpus quality, then the efficiency of corpus evaluation improves, but the complexity of the assessment system increases
Solution Approach 1:
The system uses machine learned features to enable the quality assessment system to evaluate corpora autonomously without requiring human experts to manually analyze each corpus. The pre-trained models automatically extract features related to readability, domain relevance, and content quality, allowing the system to self-assess corpus suitability. This self-service capability significantly improves evaluation efficiency while the modular architecture keeps the added complexity manageable.
Data Source
AI summary
A mechanism is provided in a data processing system for corpus quality analysis. The mechanism applies at least one filter to a candidate corpus to determine a degree to which the candidate corpus supplements existing corpora for performing a natural language processing (NLP) operation. Responsive to a determination to add the candidate corpus to the existing corpora based on a result of applying the at least one filter, the mechanism adds the candidate corpus to the existing corpora to form modified corpora. The mechanism performs the NLP operation using the modified corpora.