Deployed Medical Model Management Through Active Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Obtaining high-quality training data for medical machine learning models is costly and difficult due to the need for specialized annotations, and upgrading these models post-deployment is challenging, especially in compliance with regulatory standards like FDA requirements.
Innovation Solution
A method and system for managing deployed machine learning models that trigger an upgrade process using active learning when performance falls below a threshold, selectively sampling unlabeled data for expert labeling to improve model quality with minimal effort.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional supervised learning approaches are used with crowd-sourced annotations, then model training can be performed, but the cost and complexity increase significantly due to the need for high-quality specialized annotations in the medical domain
Solution Approach 1:
The system performs preliminary actions by pre-training a model on available data and using it to pre-label medical images before human annotation. This preliminary model generation reduces the annotation burden by providing initial labels that only require human verification and refinement, thereby reducing the quantity and cost of expert annotations needed.
Solution Approach 2:
An intermediary automated labeling system is introduced between the raw medical data and the final training data. This intermediary system uses pre-trained models to generate initial labels, which then serve as a foundation for human annotators to refine, thereby reducing the direct requirement for expensive expert annotations while maintaining data quality.
2Reliability
If models are updated after deployment to improve performance, then model accuracy can be enhanced, but the process becomes difficult and complex due to regulatory compliance requirements
Solution Approach 1:
The system performs preliminary validation and testing of model updates using active learning identified samples before full deployment. By pre-identifying which samples need retraining through performance monitoring, the system can prepare targeted update datasets in advance, simplifying the compliance approval process while maintaining model improvement.
Solution Approach 2:
A feedback mechanism is implemented where model performance is continuously monitored and used to identify samples that need retraining. This feedback loop enables targeted model updates based on actual performance degradation, making the update process more systematic and compliant with regulatory requirements while maintaining high model reliability.
3Measurement precision
If expert annotations are obtained for all data to ensure high quality, then training data quality improves, but the time and resources required increase significantly
Solution Approach 1:
Instead of requiring expert annotations for all data, the system applies partial action by using automated pre-labeling for the majority of data while reserving expert human annotation only for a subset of samples that require verification and refinement. This partial expert annotation approach maintains high data quality while significantly reducing the time and resources required compared to full expert annotation.
Solution Approach 2:
The system enables self-service by using the pre-trained model to automatically generate initial labels for training data, reducing the dependency on expert human annotation. The model serves itself by continuously improving through active learning on identified samples, thereby maintaining high data quality while minimizing the time and resources consumed for manual annotation.
Data Source
AI summary
A computer implemented method of managing a first model that was trained using a first machine learning process and is deployed and used to label medical data. The method comprises determining (202) a performance measure for the first model, and if the performance measure is below a threshold performance level, triggering (204) an upgrade process wherein the upgrade process comprises performing further training on the first model to produce an updated first model, wherein the further training is performed using an active learning process wherein training data for the further training is selected from a pool of unlabeled data samples, according to the active learning process, and sent to a labeler to obtain ground truth labels for use in the further training.