Extraction Model Training with Predictive Annotation Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for checking annotation quality in large datasets are impractical, time-consuming, and prone to errors, leading to high bias/variance complexity and incorrect annotations that affect machine learning model performance.
Innovation Solution
A computer-implemented method that generates quality scores for annotated documents, ranks them for review, and updates low-quality documents to meet a threshold before training an extraction machine learning model, using techniques like base scoring, pattern detection, and semantic analysis to improve annotation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If every annotation in a large dataset is manually checked for correctness, then annotation quality is improved, but time consumption and error-proneness increase significantly
Solution Approach 1:
The patent segments the annotation checking process into two parts: automated quality scoring using machine learning models for initial assessment, and selective manual review only for annotations that fall below quality thresholds. This segmentation resolves the contradiction by automating the majority of checks while reserving human effort for edge cases.
Solution Approach 2:
The patent implements self-service through automated quality scoring systems that independently evaluate annotation correctness without requiring manual intervention for every annotation. The system uses trained machine learning models to autonomously assess quality, reducing time consumption while maintaining reliability.
2Reliability
If multiple reviewers check annotations to improve accuracy, then annotation reliability is improved, but the complexity of the review process increases
Solution Approach 1:
The patent applies partial action by having the machine learning model perform the initial quality assessment for all annotations, and only invoking multiple human reviewers for annotations that exceed a certain error threshold. This partial application of manual review maintains reliability while reducing overall process complexity.
3Measurement precision
If all annotated documents are reviewed before training, then data extraction accuracy is improved, but productivity decreases due to time constraints
Solution Approach 1:
The patent applies local quality by differentiating the treatment of annotations based on their quality scores. High-quality annotations (above threshold) are used directly for training, while low-quality annotations undergo additional review. This localized quality control maintains extraction accuracy while improving training efficiency by avoiding unnecessary review of already high-quality data.
4Productivity
If incorrect annotations are used in training, then training speed is maintained, but model performance deteriorates due to bias/variance tradeoff complexity
Solution Approach 1:
The patent implements preliminary action by performing quality scoring and filtering of annotations before the training process begins. The machine learning model pre-assesses all annotations, identifies low-quality ones, and flags them for review or exclusion. This preliminary quality control ensures that only reliable annotations enter training, maintaining model performance while preserving training speed through efficient pre-processing.
Data Source
AI summary
Provided are techniques for training and using an extraction machine learning model based on predicting annotation quality. A first overall quality score is generated for annotated documents. It is determined that the first overall quality score is below a quality threshold. A ranked list of annotated documents is generated for review. It is determined that one or more of the annotated documents in the ranked list of annotated documents have been updated. A second overall quality score is generated for the annotated documents. It is determined that the second overall quality score is above the quality threshold. An extraction machine learning model is trained with the annotated documents. The extraction machine learning model is used to extract data items from the annotated documents.