Drifted Record Pruning for Machine Learning Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models experience performance degradation over time due to distribution shifts in data characteristics, making it inefficient for data scientists to manually relabel millions of drifted records for retraining, which is time-consuming and costly.
Innovation Solution
A method and system for managing drifted records in machine learning models by determining and pruning a subset of records for human relabeling using model confidence distributions and feature importance vectors, focusing on categories or intervals with low occurrence in training data, to improve model performance efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all drifted records are manually relabeled for retraining, then model performance can be improved, but the time and cost required becomes excessive
Solution Approach 1:
The patent segments the large set of drifted records into smaller subsets based on drift severity and category characteristics. Instead of relabeling all drifted records uniformly, the system identifies and prioritizes specific segments (e.g., records with high drift scores, records in underrepresented categories) for manual relabeling, thereby reducing the overall volume of work while maintaining model performance improvements.
Solution Approach 2:
The patent applies partial action by selecting only a proportionate subset of drifted records for manual relabeling rather than processing the entire set. The system determines an optimal number of records to relabel based on drift analysis, ensuring sufficient action to improve model performance without the excessive time cost of complete relabeling.
2Measurement precision
If a large number of drifted records are relabeled, then model accuracy improves, but the expense increases significantly
Solution Approach 1:
The patent segments drifted records into priority groups based on their impact on model performance and category representation. By focusing relabeling efforts on high-priority segments (such as records with severe drift or from underrepresented categories), the system achieves accuracy improvements at lower expense compared to relabeling all drifted records uniformly.
Solution Approach 2:
The patent changes the parameter of record selection from uniform random sampling to drift-based prioritization. By using drift scores and category distribution analysis as selection parameters, the system identifies the most critical records for relabeling, optimizing the balance between accuracy improvement and expense reduction.
3Reliability
If manual relabeling is performed on all drifted records, then comprehensive model retraining is achieved, but operational efficiency decreases
Solution Approach 1:
The patent applies partial action by performing manual relabeling on only the necessary subset of drifted records rather than the complete set. The system calculates an optimal relabeling threshold that maintains adequate model retraining while significantly improving operational efficiency by reducing the manual workload.
Solution Approach 2:
The patent implements self-service through automated drift detection and record prioritization systems that identify which records require manual relabeling. This automation reduces the burden on operators and improves operational efficiency while still achieving comprehensive retraining of critical model segments.
Data Source
AI summary
A machine learning model is trained. A set of drifted records are determined during payload time of the machine learning model. The determined set of drifted records are pruned to use in retraining of the machine learning model.