Prediction Reliability Assessment via Model Similarity Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learned and deep learned models struggle with interpretability and explainability, and they are poor at quantifying predictive uncertainty, often being overconfident in their predictions.
Innovation Solution
A computerized method that determines the reliability of a prediction output from a machine-learned or deep-learned model by using a similarity scoring method to compare outputs from multiple models with different architectures, allowing for the estimation of reliability based on epistemic and random uncertainty.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learned or deep learned models are used for prediction, then prediction accuracy is improved, but interpretability and explainability deteriorate
Solution Approach 1:
The patent introduces an intermediary reliability determination module that compares predictions from multiple models with different architectures. This mediator does not require understanding the internal workings of the black box models but instead uses their agreement/disagreement as a proxy for reliability, thus preserving prediction accuracy while providing interpretability through the comparison mechanism.
Solution Approach 2:
The system implements feedback by using the similarity between predictions from different models as a reliability indicator. This feedback loop allows the system to self-assess the trustworthiness of predictions without needing to interpret the internal state of the machine learned models, maintaining both accuracy and interpretability.
2Measurement precision
If machine learned or deep learned models are used for prediction, then prediction performance is improved, but predictive uncertainty quantification deteriorates
Solution Approach 1:
The patent uses an intermediary approach where the reliability determination module acts as a mediator between multiple model predictions and the final uncertainty assessment. Instead of relying on the models' own uncertainty estimates (which are poor), the system uses the agreement among diverse models as an intermediary signal for reliable uncertainty quantification.
Solution Approach 2:
The system changes the parameter used for uncertainty quantification from the models' internal confidence scores (which are overconfident) to the similarity metric between predictions from different models. This parameter change enables accurate uncertainty estimation while preserving the high prediction performance of the machine learned models.
3Reliability
If multiple prediction models with different architectures are used, then reliability determination is improved, but device complexity increases
Solution Approach 1:
The patent applies partial action by using a limited set of diverse models (not exhaustively all possible architectures) to determine reliability. The system uses enough model diversity to capture different biases and perspectives, but not so many that the complexity becomes unmanageable, striking a balance between reliability improvement and complexity control.
Solution Approach 2:
The reliability determination module serves multiple functions: it compares predictions from different models, calculates similarity metrics, determines reliability scores, and can detect overfitting. This multi-functionality reduces the need for separate specialized components, thereby managing complexity while achieving comprehensive reliability assessment.
4Reliability
If similarity scoring method is used to compare model outputs, then reliability assessment is improved, but computational cost increases
Solution Approach 1:
The patent uses cheap, computationally inexpensive similarity scoring methods (such as cosine similarity or simple probability comparisons) instead of expensive complex metrics. These lightweight comparison operations provide sufficient reliability assessment without incurring high computational costs, effectively using disposable low-cost operations to achieve the assessment goal.
Data Source
AI summary
A computerized method for determining the reliability of a prediction output comprises a prediction module implementing at least one first prediction model to obtain a first prediction output and at least one other prediction model to obtain another prediction output where the method further comprises a similarity module implementing a similarity scoring method to determine a similarity score between the first prediction output and the other prediction output of the at least one other prediction model, the method further comprising a reliability module implementing a reliability model which determines the reliability of the first prediction output based on the determined similarity score.