Classifier-Based Data Drift Detection for Model Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in detecting and correcting data drift, particularly concept drift, which affects the relationship between input variables and target variables over time, leading to degraded performance without clear insights into data changes or retraining needs.
Innovation Solution
A system and method for detecting drift by partitioning data sets into classifications, training multiple classifiers, and using performance metrics like AUC to identify drift, followed by selectively adding or removing data elements to correct for drift, thereby facilitating retraining with updated data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If classical monitoring approaches are used to detect model performance degradation, then model performance can be tracked, but insight into the nature of data changes and retraining requirements is lost
Solution Approach 1:
The patent segments the monitoring task into multiple specialized classifiers, each trained to detect specific types of drift (data drift, concept drift). This segmentation allows the system to track overall model performance while simultaneously providing detailed insights into the nature of data changes through specialized detection mechanisms.
Solution Approach 2:
The patent introduces drift detection classifiers as intermediary components between the model performance monitoring system and the retraining process. These classifiers act as mediators that analyze data changes and provide structured insights about drift types, bridging the gap between performance tracking and actionable retraining decisions.
2Ease of operation
If manual data accessing, visualizing, and manipulating processes are used, then data can be examined, but the process becomes time-consuming
Solution Approach 1:
The patent implements self-service automation where drift detection classifiers automatically access, analyze, and interpret data changes without manual intervention. The system autonomously performs data examination, identifies drift types, and provides recommendations, eliminating time-consuming manual processes while maintaining comprehensive data analysis capabilities.
Solution Approach 2:
The patent replaces manual mechanical processes of data examination with automated computational classifiers. These classifiers use algorithmic mechanisms to automatically detect and analyze data drift, substituting human manual operations with efficient automated systems that provide the same or better analytical capabilities without time loss.
3Measurement precision
If multiple classifiers are trained to detect different drift types, then detection accuracy improves, but system complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the drift detection system into specialized classifier modules, each responsible for detecting specific drift types. This modular segmentation improves detection precision for each drift category while organizing system complexity into manageable, independent components that can be developed and maintained separately.
Solution Approach 2:
The patent creates a universal drift detection framework where multiple specialized classifiers work together within a unified architecture. This multi-functional system handles various drift types (data drift, concept drift) through a common structure, reducing overall system complexity by providing a standardized interface and workflow that accommodates multiple detection functions.
Data Source
AI summary
Embodiments of the present disclosure include techniques for detecting and correcting drift in a data set. Data sets may be divided into classifications. A first classifier is trained on data from multiple data sets using data from each data set having a first classification. A second classifier is trained on data from the multiple data sets using data from each data set having a second classification. The performance of the classifiers are measured. Drift is detected when the performance of either classifier is above a threshold. Some embodiments may use the trained classifiers to determine data elements from one data set that are combined with another data set for training.