Meta-Model Drift Attribution for Accurate ML Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for drift detection in machine learning models lack adequate facilities for accurate determination of drift attribution, leading to suboptimal mapping between input and output drifts and poor predictive accuracy.
Innovation Solution
A meta-machine learning (meta-ML) model is trained to map input feature drift to output metric drift, using meta training data generated by computing divergences between baseline and actual input and output mappings, and employing techniques like Shapley additive explanations for attribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing drift detection methods are used, then drift detection capability is provided, but drift attribution accuracy is insufficient
Solution Approach 1:
The patent introduces a meta-ML model as an intermediary between the original ML model and the drift analysis system. This meta-model learns to map input feature drifts to output metric drifts, serving as a mediator that captures the complex relationship between input and output changes. The meta-model is trained on historical data to predict how input drifts will affect output metrics, thereby improving attribution accuracy without directly modifying the original model's predictive function.
Solution Approach 2:
The patent segments the drift analysis process into distinct components: (1) computing input feature drifts from baseline distributions, (2) using the meta-ML model to predict output metric drifts based on input drifts, and (3) attributing output drifts to specific input features. This segmentation allows each component to be optimized independently, improving overall attribution accuracy while maintaining system reliability.
2Measurement precision
If comprehensive drift analysis is performed, then drift attribution is achieved, but computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-training the meta-ML model on historical data that captures the relationship between input and output drifts. Once trained, the meta-model can quickly predict output drifts from input drifts without requiring computationally intensive analysis during deployment. The baseline distributions are also pre-computed, enabling efficient drift detection when new data arrives, thus reducing real-time computational resource requirements while maintaining high attribution accuracy.
3Reliability
If drift monitoring is implemented, then model degradation is detected, but response time is delayed
Solution Approach 1:
The patent implements continuous feedback by monitoring both input feature drifts and predicted output metric drifts in real-time. The system compares current drift measurements against thresholds and provides immediate feedback when drift exceeds acceptable levels. This feedback mechanism enables timely detection of model degradation and triggers rapid mitigation actions, reducing the time loss between drift occurrence and response while maintaining reliable performance monitoring.
Data Source
AI summary
Techniques are disclosed for using a meta-machine learning (ML) model for monitoring and attribution of drift associated with an ML model. In an example method, a training module trains a meta-ML model to map an input feature drift to an output metric drift for an ML model. The meta-ML model is trained using meta training data including a number of meta training data points. Each data point includes an input feature drift value and an output metric drift value generated by determining, for a set of input features mapped to a corresponding set of output metrics, divergences between a set of baseline input features and the set of input features and between a set of baseline output metrics and the corresponding set of output metrics. An output module outputs a predicted output metric drift of the ML model for a particular input feature drift of the ML model.