Drift Detection Model Pinpoints Accuracy Drift Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models often experience accuracy drift, making it challenging to identify the source of this drift to prevent future occurrences.
Innovation Solution
A computer-implemented method that includes identifying accuracy drift in a trained model, using a drift detection model to associate data with the drift, applying this data to a decision tree to determine the feature space and specific subset causing the drift, analyzing the feature distribution to pinpoint specific features, and returning these features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a trained machine learning model is deployed for classification operations, then the model can perform automated identification and classification tasks, but the model experiences accuracy drift over time making it difficult to identify the source of performance degradation
Solution Approach 1:
The patent implements a feedback mechanism by deploying a drift detection model that continuously monitors the trained model's performance and feeds back information about accuracy drift occurrences. This feedback loop enables the system to automatically detect when accuracy drift happens and initiates the explanation process to identify the source, thereby maintaining reliability while preserving automated classification capability.
Solution Approach 2:
The patent introduces an intermediary drift detection model and explanation system that acts as a mediator between the trained model and the monitoring process. This intermediary component detects accuracy drift and provides explanations without directly interfering with the trained model's classification operations, allowing the system to maintain both productivity and reliability.
2Measurement precision
If the entire production dataset is analyzed to identify accuracy drift sources, then comprehensive drift detection is achieved, but the computational complexity and time required increases significantly
Solution Approach 1:
The patent extracts only the relevant portion of data needed for drift detection by using the drift detection model to identify specific instances where accuracy drift occurs. Instead of analyzing the entire production dataset, the system extracts and focuses only on the subset of data associated with drift occurrences, thereby maintaining detection accuracy while reducing analysis time significantly.
Solution Approach 2:
The patent segments the production dataset into relevant and irrelevant portions based on drift detection results. By dividing the data analysis task into segments - first identifying drift occurrences, then analyzing only those specific instances - the system achieves comprehensive drift detection without the computational burden of processing the entire dataset.
3Loss of information
If a detailed analysis of all features is performed to identify specific features causing accuracy drift, then the source of drift is precisely identified, but the computational resources and processing time increase
Solution Approach 1:
The patent applies local quality by performing detailed feature analysis only in the specific regions where accuracy drift is detected, rather than uniformly analyzing all features across the entire dataset. The explanation system focuses computational resources on local feature spaces associated with drift instances, efficiently identifying the specific features causing drift while minimizing overall computational resource consumption.
Data Source
AI summary
A computer-implemented method according to one embodiment includes identifying an occurrence of accuracy drift by a trained model; identifying data associated with the accuracy drift, utilizing a drift detection model (DDM) constructed for the trained model; applying the data associated with the accuracy drift to a decision tree to determine a feature space and specific subset of the data causing the accuracy drift; analyzing a distribution of features within the feature space for the specific subset of the data causing the accuracy drift to determine specific features of the data causing the accuracy drift; and returning the specific features of the data causing the accuracy drift.