Clinical Data Extraction Using ML Confidence Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for extracting clinical variable values from clinical record data are inefficient, relying heavily on manual extraction by experts, which limits the throughput and accuracy of data processing, as they can only process a small portion of available data, leaving a large amount of clinical record data unprocessed.
Innovation Solution
A hybrid approach using machine learning models to predict clinical variable values with confidence scoring, where predicted values are automatically assigned when confidence meets a threshold, and manually extracted values are used otherwise, along with a graphical user interface (GUI) to facilitate user input and restrict modifications of confidently predicted values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual extraction by experts is used, then accuracy of extracted clinical variable values is maintained, but throughput and processing speed are severely limited
Solution Approach 1:
The extraction process is segmented into two distinct paths: automated machine learning extraction for high-throughput processing and manual expert extraction for accuracy-critical cases. The system divides clinical record data into segments that can be processed by ML models and segments requiring human review, enabling parallel processing and significantly increasing overall throughput while maintaining accuracy standards.
Solution Approach 2:
A confidence scoring mechanism serves as an intermediary between automated ML extraction and manual expert review. The ML model generates confidence scores for each extracted value, and values below a threshold are automatically routed to human experts for verification. This intermediary layer enables the system to leverage both automated speed and human accuracy without requiring all data to undergo manual review.
2Productivity
If automated machine learning extraction is used, then throughput is significantly increased, but accuracy and reliability of extracted values deteriorate
Solution Approach 1:
The system implements a feedback loop where extracted values from ML models are evaluated against confidence thresholds, and values failing to meet the threshold are flagged for manual review. The results from manual review are fed back into the system to continuously improve and retrain the ML models, creating a self-improving extraction pipeline that maintains high accuracy while achieving high throughput.
Solution Approach 2:
Instead of requiring 100% manual review for accuracy, the system applies partial manual action only to cases where the ML model confidence is below the threshold. This selective approach means that the majority of high-confidence extractions are accepted automatically, achieving high throughput, while a small fraction of uncertain cases receive manual verification to maintain overall accuracy and reliability.
3Quantity of substance
If all clinical record data is processed manually, then completeness of data extraction is achieved, but the volume of data that can be processed is severely limited
Solution Approach 1:
The system performs preliminary automated extraction using ML models for all clinical record data before manual review. This preliminary action processes the entire data volume quickly, identifying high-confidence extractions that can be accepted immediately. Only the remaining uncertain cases are then subjected to manual review, dramatically reducing the time and effort required while ensuring completeness of data extraction across the entire dataset.
4Reliability
If manual extraction is used, then quality and accuracy of extracted information is maintained, but the cost and resource requirements increase
Solution Approach 1:
The system changes the parameter of expert involvement from constant (100% manual review) to variable (threshold-based selective review). By adjusting the confidence threshold parameter, the system can optimize the balance between quality and resource requirements. High-confidence extractions are accepted automatically, reducing expert workload and resource consumption, while maintaining quality standards through threshold-controlled manual review of uncertain cases.
Data Source
AI summary
Described herein are techniques of using machine learning to automatically extract clinical variable values for subjects from clinical record data. The techniques designate certain clinical variables as hybrid variables that can be assigned values by machine learning model prediction. The techniques process, using a machine learning model trained to predict a value of a hybrid variable, clinical record data associated with a subject to obtain a predicted hybrid variable value and an associated confidence score. The techniques set the value of the hybrid variable for the subject to the predicted hybrid variable value when the model prediction is of sufficiently high confidence.