Clustered Explainability Models for Real-Time ML Predictions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning model explainability techniques are computationally intensive, leading to significant computational load that exceeds available resources, especially when explaining multiple transactions in real-time.
Innovation Solution
A method that classifies input transactions into input space clusters and maps them to explainability space clusters, using interpretable models to explain predictions, and constructs an explainability cache to reduce recomputation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If current explainability techniques (LIME, SHAP) are used to explain machine learning model predictions, then model explainability and user trust are improved, but computational requirements and resource demands increase significantly
Solution Approach 1:
The patent segments the input space into multiple clusters, where each cluster is explained by a dedicated interpretable model. This divides the single complex explanation task into multiple simpler local tasks, reducing the computational burden of applying global explainability methods like LIME or SHAP to all transactions.
Solution Approach 2:
The patent performs clustering and trains interpretable models in advance before production deployment. During real-time inference, transactions are simply classified into existing clusters and explained using pre-trained models, eliminating the need for computationally intensive explainability calculations at runtime.
2Loss of information
If explainability techniques are applied to multiple transactions in real-time, then model transparency and compliance are improved, but processing latency and computational costs increase
Solution Approach 1:
The patent performs all computationally intensive operations including clustering analysis and interpretable model training during the offline preparation phase. At runtime, the system only needs to perform lightweight cluster classification and apply pre-computed explanations, dramatically reducing processing latency for real-time transactions.
Solution Approach 2:
By dividing transactions into distinct clusters with dedicated interpretable models, the system can process and explain multiple transactions in parallel using simple model applications rather than running expensive explainability algorithms on each transaction sequentially.
3Loss of information
If traditional explainability methods are used for all transactions, then comprehensive model explanation is achieved, but resource demands exceed available compute capacity
Solution Approach 1:
The patent partitions the transaction space into multiple clusters, each handled by a lightweight interpretable model. This segmentation allows comprehensive explanation coverage across all transactions while using minimal computational resources per transaction, as each cluster model is much simpler than global explainability methods.
Solution Approach 2:
The patent creates multiple copies of simple interpretable models, one for each cluster, rather than using a single complex explainability system. These copied models can be deployed distributedly and executed in parallel, providing comprehensive explanation coverage without exceeding compute capacity.
Data Source
AI summary
A first input transaction is classified into a first input space cluster in a set of input space clusters. It is determined that the first input space cluster maps to a single explainability space cluster in a set of explainability space clusters. Using an interpretable model corresponding to the single explainability space cluster, a first machine learning model prediction is explained, the first machine learning model prediction resulting from processing, by a machine learning model, the first input transaction.