ML Model Performance Monitoring via Data Drift Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques lack effective methods for monitoring the performance of machine learning models in production environments without ground truth data, making it difficult to determine when models need updating or replacing due to performance issues.
Innovation Solution
A computer-implemented method that computes a data drift score based on inputs and outputs of a machine learning model over predefined time periods, using a policy that specifies baseline and target intervals, comparison metrics, and alert thresholds to identify performance changes and generate alerts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If ground truth data is used to evaluate ML model performance, then measurement precision is improved, but availability of data is worsened
Solution Approach 1:
The patent introduces an intermediary approach by using surrogate metrics and proxy measurements instead of direct ground truth comparisons. The system employs data drift detectors that measure changes in input data distributions and model output behaviors as intermediate indicators of performance degradation, bypassing the need for direct ground truth data while still providing meaningful performance assessment
Solution Approach 2:
The patent replaces the traditional mechanical comparison mechanism (direct ground truth validation) with a statistical and information-theoretic approach. Instead of mechanically comparing predictions to known correct answers, the system uses entropy calculations, distributional comparisons, and anomaly detection algorithms to infer performance status without requiring ground truth labels
2Measurement precision
If data drift detection is performed over long time intervals, then detection of slow drifts is improved, but response time to performance issues is worsened
Solution Approach 1:
The patent implements periodic action through scheduled drift detection at multiple time intervals. The system performs both long-interval drift detection (for slow drifts) and short-interval monitoring (for rapid changes), creating a multi-timescale detection rhythm that balances detection accuracy with response time requirements
Solution Approach 2:
The patent adds another dimension to time-based detection by implementing hierarchical monitoring across different temporal scales. Instead of a single time interval, the system operates simultaneously at multiple time dimensions (short-term, medium-term, long-term), allowing it to detect both rapid and slow drifts without compromising response time
3Reliability
If comprehensive monitoring of ML model inputs and outputs is implemented, then reliability is improved, but device complexity is worsened
Solution Approach 1:
The patent applies segmentation by dividing the monitoring system into distinct modular components: data drift detectors for input features, prediction drift detectors for model outputs, entropy calculators, and alerting mechanisms. Each component handles a specific aspect of monitoring independently, reducing overall system complexity while maintaining comprehensive coverage
Solution Approach 2:
The patent implements universality through a unified drift detection framework that handles multiple types of ML models (classification, regression, clustering) and multiple data types using the same core algorithms. The entropy-based detection methods and distributional comparison techniques serve as universal tools applicable across different model types, reducing the need for model-specific monitoring complexity
Data Source
AI summary
One embodiment of a method for monitoring performance of a trained machine learning model includes receiving at least one of one or more first inputs or one or more first outputs of the trained machine learning model during a first time period, receiving at least one of one or more second inputs or one or more second outputs of the trained machine learning model during a second time period, and computing a data drift score based on the at least one of the one or more first inputs or the one or more first outputs, the at least one of the one or more second inputs or the one or more second outputs, and a predefined policy.