Discrimination Device for Automatic Annotation Error Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual annotation of images for defect detection in machine learning systems often leads to inconsistent labeling, resulting in inaccurate training models and time-consuming corrections, as subtle differences and human errors can cause erroneous determinations.
Innovation Solution
A method that involves repeatedly performing discrimination processing on learning data sets to detect annotations with high probabilities of being incorrect, using a sub-learning data set and sub-verification data set to generate a pre-trained model and calculate the probability of correctness of labels, allowing for automatic detection and correction of errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation is performed by operators to label image data, then the learning data can be created for machine learning, but annotation errors and inconsistencies occur due to human mistakes and difficulty in detecting subtle differences
Solution Approach 1:
The system uses the trained model to automatically detect and flag potential annotation errors in the learning data. The model serves itself by evaluating the quality of annotations used to train it, identifying inconsistent or erroneous labels without requiring external verification for every data point.
Solution Approach 2:
The system implements a feedback mechanism where discrimination results are used to evaluate annotation quality. When the model identifies annotations with high error probability, these are flagged for correction, creating a continuous improvement loop that enhances annotation consistency and accuracy over time.
2Measurement precision
If all annotations are visually reconfirmed by operators to correct errors, then annotation accuracy can be improved, but the process becomes extremely time-consuming
Solution Approach 1:
Instead of requiring visual reconfirmation of all annotations, the system applies partial action by only flagging specific annotations that have high error probability according to the discrimination results. This selective approach significantly reduces the time required for correction while maintaining high annotation accuracy.
Solution Approach 2:
The system automatically identifies which annotations require correction based on discrimination results, eliminating the need for operators to manually review every annotation. The self-service mechanism prioritizes only the most problematic cases for human review.
3Productivity
If the pre-trained model is trained with erroneous annotations, then the model can be generated quickly, but the discrimination accuracy and reliability of inspection results deteriorate
Solution Approach 1:
The system performs preliminary evaluation of annotation quality using discrimination results before finalizing the learning data set. By identifying and flagging erroneous annotations in advance, the system ensures that only high-quality annotations are used for training, thereby maintaining high discrimination accuracy while preserving efficient model training.
Data Source
AI summary
A discrimination device includes a sub-data set extraction unit for extracting from a plurality of labeled learning data a sub-learning data set to be used for learning and a sub-verification data set to be used for verification, a learning unit for performing supervised learning on the basis of the sub-learning data set to generate a pre-trained model for discriminating a label from data related to an object, a discrimination unit for conducting a discrimination processing using the pre-trained model on each piece of learning data contained in the sub-verification data set, a verification result recording unit for recording a result of the discrimination processing in association with the learning data, and a correctness detection unit for detecting learning data attached with a label that may be incorrect based on the discrimination processing results recorded in association with respective learning data.