Clinical Diagnostic Model Evaluation Metric
Solution Overview
Problem
Current metrics for evaluating multi-label multi-class computational diagnostic models are inadequate as they do not align with clinical principles, fail to differentiate between wrong, missed, and over diagnoses, and are influenced by dataset prevalence, making it challenging to compare model efficacy and choose the best diagnostic model for clinical applications.
Innovation Solution
A method and system that classify predicted diagnoses into wrong, missed, over, or right diagnoses, calculate penalties based on a contradiction matrix, and compute a score using a metric that accounts for perfect and null model scores to evaluate the clinical efficacy of multi-label multi-class computational diagnostic models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional machine learning metrics (Hamming Loss, subset accuracy, etc.) are used to evaluate multi-label multi-class computational diagnostic models, then the evaluation can be performed using standard metrics, but the metrics do not align with clinical principles and fail to differentiate between wrong, missed, and over diagnoses
Solution Approach 1:
The patent transforms the evaluation approach by changing the parameters used in metric calculation. Instead of using conventional machine learning metrics, it introduces a clinically-aligned metric that incorporates penalty weights for different diagnosis types (wrong diagnosis penalty wwd, missed diagnosis penalty wmd, over diagnosis penalty wod) and contradiction penalties based on medical knowledge. This parameter transformation enables the metric to reflect clinical principles while maintaining computational evaluability.
2Adaptability or versatility
If different metrics are used to evaluate diagnostic models, then various aspects of performance can be measured, but the metrics do not agree on comparative performance and model selection becomes metric-dependent
Solution Approach 1:
The patent merges multiple evaluation considerations into a single unified metric. It combines the multi-label classification performance measurement with clinical principle adherence by integrating penalty weights for different diagnosis types and contradiction penalties into one comprehensive metric formulation. This unification ensures consistent model comparison while maintaining versatility in measuring various performance aspects.
Solution Approach 2:
The proposed metric serves multiple functions simultaneously: it evaluates classification accuracy, penalizes different types of errors according to clinical principles, incorporates medical knowledge through contradiction penalties, and enables consistent model comparison. This multi-functionality makes the metric universally applicable for evaluating multi-label multi-class computational diagnostic models across different clinical contexts.
3Reliability
If sensitivity is increased to reduce missed diagnoses, then more ailments are detected, but false positives increase leading to alarm fatigue
Solution Approach 1:
The patent applies local quality by assigning different penalty weights to different types of diagnostic errors based on their clinical significance. Instead of treating all errors uniformly, it applies weighted penalties: wwd for wrong diagnoses, wmd for missed diagnoses, and wod for over diagnoses. This localized penalty approach allows the metric to account for the varying harm caused by different error types, enabling balanced optimization of sensitivity and specificity according to clinical priorities.
Solution Approach 2:
The metric incorporates asymmetry by treating different diagnosis errors differently through unequal penalty weights. The penalty for missed diagnoses (wmd) can be set higher than for over diagnoses (wod) when sensitivity is prioritized, or vice versa when reducing false positives is more important. This asymmetric treatment allows flexible alignment with clinical principles while maintaining metric coherence.
4Productivity
If conventional metrics are used, then evaluation is simple and fast, but the metrics are influenced by dataset prevalence and cannot reliably compare model efficacy
Solution Approach 1:
The patent applies preliminary action by pre-defining penalty weights (wwd, wmd, wod) and contradiction penalties based on clinical knowledge before model evaluation. The contradiction matrix and penalty parameters are established in advance according to medical principles, allowing the metric to quickly evaluate models without being influenced by dataset prevalence. This pre-configured approach maintains evaluation speed while improving measurement precision.
Data Source
AI summary
The present invention relates to the field of evaluating clinical diagnostic models. Conventional metrics does not consider context dependent clinical principles and is unable to capture critically important features that ought to be present in a diagnostic model. Thus, present disclosure provides a method and system for evaluating clinical efficacy of multi-label multi-class computational diagnostic models. Diagnosis for a given dataset of diagnostic samples is obtained from the diagnostic model which is then classified as wrong, missed, over or right diagnosis, based on which a first penalty is calculated. A second penalty is calculated for each diagnostic sample using a contradiction matrix. The first and second penalties are summed up to compute a pre-score for each diagnostic sample. Finally, the diagnostic model is evaluated using a metric that is based on sum of pre-scores, and scores from a perfect and a null multi-label multi-class computational diagnostic model.