Classifier-Based Data Drift Detection for Targeted Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in detecting and correcting data drift, which occurs when training and validation data sets change over time, leading to concept drift that affects model performance, and retraining with new data is technically challenging.
Innovation Solution
A system and method for detecting drift by partitioning data sets into classifications, training multiple classifiers, measuring their performance using AUC, and selectively adding data elements to the newer data set to correct for drift, ensuring effective retraining without introducing further drift.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If classical monitoring approaches are used to detect data drift, then model performance trends can be observed, but little insight is provided about the nature of data changes and how to retrain the model
Solution Approach 1:
The patent segments the data set into multiple subsets using clustering algorithms (e.g., K-means). Each subset represents a distinct data distribution or concept. This segmentation allows the system to identify which specific subsets are causing performance degradation and provides actionable insights about the nature of data drift, enabling targeted retraining strategies.
2Measurement precision
If manual data accessing, visualizing, and manipulating processes are used, then data can be examined, but the process is time consuming
Solution Approach 1:
The system performs automated data subset identification, drift detection, and retraining recommendations without requiring manual intervention. The clustering algorithms automatically segment the data, identify drifting subsets, and suggest which subsets should be included or excluded from retraining. This self-service approach eliminates time-consuming manual processes while maintaining high measurement precision.
3Adaptability or versatility
If the model is retrained with all new data to address drift, then the model may adapt to new patterns, but further drift may be introduced
Solution Approach 1:
The patent applies local quality by treating different data subsets differently during retraining. Instead of uniformly including or excluding all new data, the system identifies specific subsets that exhibit drift and targets only those for exclusion or modified inclusion. This localized approach allows the model to adapt to beneficial new patterns while excluding harmful drifted subsets, maintaining reliability while improving adaptability.
4Measurement precision
If multiple classifiers are trained to detect drift, then detection accuracy improves, but the system complexity increases
Solution Approach 1:
The patent merges multiple clustering algorithms and drift detection methods into a unified framework. Instead of running separate independent systems for each classifier, the system integrates them to work together synergistically. The clustering results from multiple algorithms are combined to identify consensus data subsets, and drift detection is performed across the integrated structure. This merging reduces overall system complexity while maintaining or improving detection accuracy through the combined strength of multiple approaches.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
Embodiments of the present disclosure include techniques for detecting and correcting drift in a data set. Data sets may be divided into classifications. A first classifier is trained on data from multiple data sets using data from each data set having a first classification. A second classifier is trained on data from the multiple data sets using data from each data set having a second classification. The performance of the classifiers are measured. Drift is detected when the performance of either classifier is above a threshold. Some embodiments may use the trained classifiers to determine data elements from one data set that are combined with another data set for training.