File Annotation Model Retraining via Confidence-Based Target Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file annotation methods rely heavily on manual truth-value annotation information, which is costly and inefficient, especially when using machine learning models that require extensive manual labeling to achieve accurate results.
Innovation Solution
A method that processes files using an annotation model to determine its performance based on confidence levels, identifies target files for retraining when performance is below a threshold, and iteratively improves the model's performance by acquiring truth-value annotation information for these files, thereby reducing the need for extensive manual annotation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation is used to train the annotation model, then the model performance improves, but the cost and time consumption increase significantly
Solution Approach 1:
The patent applies partial action by selecting only a subset of files for manual annotation rather than annotating all files. The system identifies files with low confidence scores from the model's predictions and annotates only those, rather than requiring manual annotation of the entire dataset. This reduces the time and cost of manual annotation while still improving model performance through targeted retraining.
2Measurement precision
If more truth-value annotation information is collected for retraining, then the model accuracy improves, but the annotation cost increases
Solution Approach 1:
The patent segments the annotation process into two distinct phases: initial model training with a small set of manually annotated files, and iterative improvement phases where the model processes files, identifies low-confidence predictions, and selectively annotates only those segments. This segmentation allows the system to achieve high accuracy without incurring the cost of annotating the entire file set.
Solution Approach 2:
The annotation model performs self-service by automatically identifying which files require manual annotation based on its confidence scores. The system autonomously determines that files with low confidence predictions should be selected for manual annotation, eliminating the need for manual review of all files and significantly reducing annotation costs while maintaining accuracy improvement.
3Reliability
If the annotation model is retrained with more data, then the confidence of predictions improves, but the processing time increases
Solution Approach 1:
The patent applies partial action in the retraining process by using only the subset of files that were identified as needing annotation (those with low confidence scores). Rather than retraining on the entire dataset, the system retrains only on the selectively annotated portion, which maintains prediction confidence improvement while significantly reducing the processing time required for retraining.
Data Source
AI summary
File annotation is described. An example method includes: processing files to be annotated by using an annotation model to determine a first performance of the annotation model, the first performance being associated with the confidence of a model annotation result generated by the annotation model; if the first performance is lower than a predetermined threshold, determining a group of target files from the files based at least on the confidence of the model annotation result; acquiring truth-value annotation information of the group of target files for retraining the annotation model; and if a second performance of the retrained annotation model is higher than or equal to the predetermined threshold, determining annotation information for at least some of the files by using the retrained annotation model. Based on this approach, automatic annotation of files can be realized with less truth-value annotation information, thereby reducing annotation costs.