Medical Image Annotation Quality Evaluation Using Reference History
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for medical image analysis lack an effective method to accurately evaluate the quality of annotation information, which is crucial for their performance.
Innovation Solution
An information processing apparatus and method that acquires reference history information from annotators regarding related medical information to derive evaluation information for annotation quality, enhancing the accuracy of annotation information used in machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If annotation information is generated by annotators for machine learning models, then the machine learning model can be trained and evaluated, but the quality and reliability of the annotation information cannot be accurately evaluated
Solution Approach 1:
The system implements feedback by acquiring reference history information about the annotator's past work and using it to evaluate the quality of current annotation information. The evaluation information is then used to assess reliability, creating a closed-loop system where past performance informs future quality assessment
Solution Approach 2:
The system performs preliminary action by acquiring reference history information before the actual annotation task is completed. This pre-existing data about the annotator's history with similar medical images is used to predict and evaluate the quality of the annotation information before it is fully processed into the machine learning model
2Reliability
If comprehensive evaluation of annotation quality is implemented, then the reliability of machine learning models improves, but the complexity of the evaluation system increases
Solution Approach 1:
The system extracts only the necessary reference history information related to the annotator's experience with similar medical images, rather than evaluating all possible annotator attributes. This selective extraction simplifies the evaluation system while maintaining reliability assessment accuracy
Solution Approach 2:
The evaluation system focuses on local quality by assessing specific aspects of annotator competence relevant to the particular medical image type and annotation task, rather than requiring comprehensive evaluation of all annotator skills. This targeted approach reduces system complexity while improving relevance
Data Source
AI summary
An information processing apparatus includes a processor, in which the processor acquires, when an annotator generates annotation information as correct answer data of a machine learning model for analyzing a medical image, reference history information by the annotator of related medical information related to a generation source medical image that is a generation source of the annotation information, and derives evaluation information representing quality of the annotation information based on the reference history information.