Annotation Label Display Using Likelihood Scores to Reduce Mislabeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing annotation tools for machine learning may inadvertently include irrelevant labels in learning data due to incorrect candidate presentation, leading to degraded prediction accuracy in machine learning models.
Innovation Solution
An information processing apparatus that estimates candidate labels using a trained model, controls display positions based on likelihood scores, and provides differentiated display information to assist users in selecting appropriate labels, thereby reducing incorrect selections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If candidates for labels are extracted and presented to users using a trained model, then the work load on users is reduced, but irrelevant labels may be included in learning data due to incorrect candidate presentation
Solution Approach 1:
The patent applies local quality by differentiating the display presentation of candidate labels based on their individual likelihood scores. High-scoring candidates receive prominent display positions while low-scoring candidates are displayed less prominently or with indicators suggesting lower reliability. This allows the system to maintain automated candidate extraction for productivity while ensuring data reliability through differentiated presentation that guides user selection.
Solution Approach 2:
The patent changes the display parameters (position, size, visual emphasis) of candidate labels based on their likelihood scores from the trained model. By dynamically adjusting these presentation parameters, the system maintains the efficiency of automated candidate generation while improving data accuracy through score-based differentiation in the user interface.
2Measurement precision
If multiple candidate labels are displayed to users for selection, then the accuracy of label selection can be improved, but the complexity of the annotation tool increases
Solution Approach 1:
The system performs preliminary action by using the trained model to pre-analyze target data and extract candidate labels with calculated likelihood scores before presenting them to users. This preliminary processing reduces the complexity of the annotation task while maintaining selection accuracy, as users only need to review pre-processed candidates rather than creating labels from scratch.
Solution Approach 2:
The patent uses copying by having the trained model generate candidate labels that replicate or approximate the correct annotations. These copied candidates are then presented to users for verification, maintaining accuracy while simplifying the user's workload compared to creating annotations manually.
3Productivity
If automated extraction of label candidates is used, then productivity is improved, but the quality of learning data may degrade due to incorrect selections
Solution Approach 1:
The patent implements feedback by using the likelihood scores from the trained model to guide the presentation of candidate labels to users. Users can see which candidates have higher confidence scores and make more informed selections. This feedback mechanism maintains the productivity benefits of automated extraction while improving data quality through score-informed user decision-making.
Solution Approach 2:
The likelihood score acts as an intermediary between the automated extraction process and the final label selection. It mediates by providing quantitative information about candidate quality, allowing the system to maintain automation efficiency while ensuring data quality through the intermediary scoring mechanism that guides user choices.
Data Source
AI summary
An information processing apparatus is provided and includes an obtaining unit obtains an estimation result of a score representing a likelihood for each of a first candidate and a second candidate for a label to be added as an annotation to data to be annotated. A control unit controls processing for displaying, depending on the score for each of the first candidate and the second candidate, first display information and second display information through an output unit, the first display information indicating a display position associated with the first candidate, the second display information indicating a display position associated with the second candidate.