Annotation Quality Feedback Using Embedded Ground Truth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for creating training data for machine learning models often result in poor annotation accuracy, with vendors delivering low-quality training data, necessitating a method to ensure the quality of manual processing results on multiple targets.
Innovation Solution
An information processing apparatus that evaluates annotation accuracy periodically during data creation by comparing operator inputs with ground truth data, allowing for immediate detection of quality deterioration and enabling correction, ensuring high-quality training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual processing is performed on multiple processing targets, then productivity increases, but manufacturing precision deteriorates due to poor annotation accuracy
Solution Approach 1:
The system performs preliminary actions by embedding ground truth processing targets within the batch of processing targets before manual processing begins. This allows quality reference data to be prepared in advance, enabling subsequent comparison and evaluation without delaying the manual processing workflow.
Solution Approach 2:
The system implements feedback by calculating evaluation values that compare operator processing results against ground truth results for ground truth processing targets. This feedback mechanism provides real-time quality assessment, allowing operators to correct errors and managers to monitor annotation accuracy while maintaining high processing throughput.
2Manufacturing precision
If quality evaluation is performed continuously, then manufacturing precision is maintained, but device complexity increases
Solution Approach 1:
The information processing apparatus performs multiple functions using a unified system: it manages batch processing workflows, embeds ground truth targets, collects operator results, calculates evaluation values through comparison, and provides quality feedback. This multi-functional approach maintains annotation quality without requiring separate complex quality control systems.
Data Source
AI summary
An information processing apparatus includes: an acceptance unit that accepts processing results of a plurality of processing targets including ground truth processing targets associated with ground truth processing results every time an operator completes processing on part of the processing targets; and a calculation unit that, when processing targets associated with the processing results accepted by the acceptance unit are the ground truth processing targets, calculates an evaluation value indicating an evaluation on the operator based on comparison between the processing results accepted by the acceptance unit and the ground truth processing results.