Medical Data Annotation Selection Using AI Inference Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The annotation of medical data for AI model training is costly and time-consuming, and randomly selecting data can lead to suboptimal training performance due to budget limitations.
Innovation Solution
A method and apparatus for selecting medical data for annotation based on the current training performance of an AI model, determining a selection policy to identify target data for next training, and using inference results to intelligently select data for annotation, potentially adding randomly extracted data for diversity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all medical data is annotated to improve AI model training performance, then training performance is improved, but annotation costs and time increase significantly
Solution Approach 1:
The patent segments the annotation process into multiple stages: initial random sampling for baseline training, then iterative selection of specific data subsets based on model performance metrics. Instead of annotating all data at once, the system divides the large-scale annotation task into manageable segments that can be processed sequentially based on priority and need.
Solution Approach 2:
The system changes the selection parameter from random sampling to performance-based sampling. By monitoring model accuracy, loss functions, and other performance parameters, the system dynamically adjusts which data subsets should be annotated next, focusing resources on data that will most improve model performance.
2Loss of time
If data is randomly extracted for annotation to reduce costs, then annotation costs are reduced, but training performance may be suboptimal
Solution Approach 1:
The system implements a feedback loop where model performance on validation data continuously informs the selection of next annotation targets. The model's predictions, confidence scores, and error patterns provide feedback that guides which uncertain or problematic cases should be annotated next, ensuring each annotation round addresses specific performance gaps.
Solution Approach 2:
The system performs preliminary random annotation to establish a baseline model before transitioning to targeted annotation. This preliminary action creates an initial model that can then identify its own weaknesses and guide subsequent annotation efforts, rather than starting directly with targeted selection.
3Manufacturing precision
If skilled experts annotate medical data to ensure quality, then annotation quality is improved, but costs increase significantly
Solution Approach 1:
The system applies partial annotation action by selecting only the most critical data subsets that will yield the greatest performance improvement. Instead of having experts annotate all data uniformly, the system identifies specific high-value targets (e.g., ambiguous cases, performance-critical regions) that require expert attention, while other data can be handled differently or deferred.
Data Source
AI summary
An operating method of a medical data selecting apparatus operated by at least one processor includes generating training data including partial medical data sampled from mass medical data and annotated data of the partial medical data, extracting candidate data for annotation from the mass medical data, the candidate data being at least a portion of the mass medical data, acquiring inference results that are inferred from the candidate data by an artificial intelligence (AI) model trained based on the training data and selecting target data for annotation to be used in next training of the AI model, from among the candidate data based on the inference results.