Data Drift Detection via Weighted Outlier Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for detecting dataset shift between training and test data in machine learning are too sensitive and prone to false alarms, particularly in large data environments, and struggle with high-dimensional datasets, often rejecting the null hypothesis of no difference even when changes are minor or inconsequential.
Innovation Solution
A computer-implemented method using a pooled dataset to train a scoring function, calculate outlier scores, assign weights based on contamination rates, and determine a test statistic, specifically the weighted area under the receiver characteristic curve (WAUC), to detect dataset shift by focusing on low-density regions and discounting inliers, thereby reducing false alarms and handling high-dimensional data effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing techniques for detecting data drift are used, then detection sensitivity is improved, but false alarm rate increases
Solution Approach 1:
The patent applies local quality by differentiating the treatment of different data regions through contamination rates. Inlier regions (high density) are discounted with lower weights, while outlier regions (low density) are emphasized with higher weights. This selective weighting based on local data characteristics resolves the contradiction by making the detection sensitive to meaningful drift in critical regions while being robust to noise in typical regions.
Solution Approach 2:
The patent changes the parameter of data point weighting based on contamination rates derived from density estimation. By transforming the uniform treatment of all data points into a differentiated weighting scheme where weights are functions of local density, the method achieves both sensitivity to important drift and robustness to false alarms through parameter adaptation.
2Quantity of substance
If existing techniques are applied to large data sets, then detection coverage is improved, but computational complexity increases
Solution Approach 1:
The patent extracts only the essential information needed for drift detection by using density estimation to compute contamination rates, rather than performing complex comparisons between entire high-dimensional distributions. This extraction of key characteristics (density-based weights) from the full data enables scalable detection on large data sets without prohibitive computational complexity.
3Measurement precision
If existing techniques are used on high-dimensional datasets, then detection capability is improved, but false positives increase
Solution Approach 1:
The patent addresses the curse of dimensionality by focusing detection on local data density characteristics rather than global distribution properties. The contamination rate-based weighting emphasizes regions where actual drift is likely to occur (low-density outlier regions) while discounting high-density inlier regions that are prone to false positives in high-dimensional spaces.
Data Source
AI summary
Data drift or dataset shift is detected between training dataset and test dataset by training a scoring function using a pooled dataset, the pooled dataset including a union of the training dataset and the test dataset; obtaining an outlier score for each instance in the training dataset and the test dataset based at least in part on the scoring function; assigning a weight to each outlier score based at least in part on training contamination rates; determining a test statistic based at least in part on the outlier scores and the weights; determining a null distribution of no dataset shift for the test statistic; determining a threshold in the null distribution; and when the test statistic is greater than or equal to the threshold, identifying dataset shift between the training dataset and the test dataset.