AI Classifier Risk Metric Selection for Misclassification Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is no standard method for assessing the risk of misclassification inherent in using artificial intelligence (AI) or machine learning (ML) for feature classification, which is crucial for applications like fault classification on turbine blades.
Innovation Solution
A computer-implemented method and apparatus for initiating the use of classifier models by obtaining classifier confusion information and risk confusion information, determining a risk metric for each model, and selecting models based on these metrics to manage the risk of misclassification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a fully autonomous ML solution is used to perform feature classification, then productivity is improved by replacing manual visual inspection, but reliability deteriorates due to the risk of misclassification
Solution Approach 1:
The patent implements feedback mechanisms by calculating risk metrics from confusion matrices and using them to adjust classifier operation. The system continuously monitors misclassification patterns and feeds this information back to modify classification thresholds or trigger re-evaluation, thereby maintaining high productivity while improving reliability through adaptive correction of errors.
Solution Approach 2:
The patent performs preliminary actions by generating confusion matrices and calculating risk metrics before final classification decisions are made. This pre-assessment of potential misclassification risks allows the system to adjust parameters or flag high-risk cases beforehand, ensuring both efficient processing and reliable outcomes.
2Reliability
If classification thresholds are lowered to reduce false negatives, then reliability is improved for critical faults, but object-generated harmful factors increase due to false positives
Solution Approach 1:
The patent applies local quality by implementing different classification thresholds and risk tolerances for different fault types. Critical faults with high safety implications use lower thresholds to ensure detection, while less critical faults use higher thresholds to minimize false alarms. This localized adjustment of classification stringency optimizes both detection sensitivity and false positive rates for different classes.
Solution Approach 2:
The patent dynamically changes classification parameters based on risk metrics calculated from confusion matrices. By adjusting decision thresholds, confidence levels, and risk weights according to the specific fault type and operational context, the system achieves high reliability for critical detections while controlling false positive rates through parameter optimization.
3Reliability
If multiple classifier models are evaluated and selected based on risk metrics, then reliability is improved through better model selection, but device complexity increases
Solution Approach 1:
The patent applies partial action by evaluating and maintaining multiple classifier models, but only activating or prioritizing those that meet specific risk metric thresholds. This approach ensures high reliability through comprehensive model evaluation while controlling complexity by not deploying all possible models simultaneously, but rather selecting the appropriate subset based on performance and risk criteria.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments described herein relate to methods and apparatuses for performing classification of a classification subject. A computer implemented method for initiating use of one or more classifier models to perform classification of a classification subject into one of a plurality of classes comprises: for each of a plurality of classifier models, obtaining respective classifier confusion information, wherein the classifier confusion information indicates, for each actual class in the plurality of classes, the likelihood of the classifier model classifying the actual class as each predicted class in the plurality of classes; obtaining risk confusion information indicating a risk associated with each actual class and predicted class pair; for each of the plurality of classifier models, determining a respective risk metric from the respective classifier confusion information and the risk confusion information; and initiating use of one or more of the plurality of classifier models based on the respective risk metrics.