Drift Mode Acquisition via Successive Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models experience performance deterioration over time due to changes in input distribution or mapping, which can be misinterpreted as normal performance, leading to incorrect decision-making and potential chain reactions.
Innovation Solution
Implementing recurrent training of overfitted sub-models to inform drift detection methods and determine when the main model should be retrained or replaced, by constructing drift mode curves from absolute difference comparisons between reference and subsequent sub-models trained at intervals, allowing for identification of drift modes such as sudden, gradual, incremental, and recurring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If drift detection is performed using the main model, then detection accuracy is improved, but computational resources and time are consumed
Solution Approach 1:
The patent creates a simplified copy of the main model called a 'drift model' that replicates only the essential drift detection functionality. This copy is trained on historical drift data and can quickly identify drift modes without requiring the full computational power of the main model, thus reducing detection time while maintaining accuracy.
Solution Approach 2:
The drift detection system is segmented into separate functional components: the main model for production inference and the drift model for detection. The drift model is further divided into specialized modules that handle different drift detection tasks independently, allowing for more efficient processing and reduced computational overhead.
2Reliability
If the main model is continuously monitored for drift, then reliability is improved, but system complexity increases
Solution Approach 1:
The drift model acts as an intermediary between the main model and the monitoring system. Instead of directly monitoring the complex main model, the system monitors the simplified drift model which captures drift characteristics. This intermediary layer simplifies the monitoring architecture while maintaining reliable drift detection.
Solution Approach 2:
The drift model is trained using historical drift data from the main model's own operations, allowing it to self-monitor and identify drift patterns. This self-service capability reduces the need for external complex monitoring systems and manual intervention, simplifying the overall architecture while maintaining reliability.
3Use of energy by stationary object
If drift detection uses simplified models, then computational resources are reduced, but detection precision may deteriorate
Solution Approach 1:
The drift model uses different parameter settings and training approaches optimized for detection rather than production performance. It is trained on historical drift data with a focus on capturing drift patterns, using parameters tuned for detection precision rather than prediction accuracy, thus achieving high detection precision with lower computational resources.
Solution Approach 2:
The drift model is pre-trained on historical drift data before actual drift detection is needed. This preliminary training allows the model to learn drift patterns in advance, so that during actual drift detection, it can quickly and accurately identify drift modes without requiring complex real-time computation, thus reducing computational resources while maintaining precision.
Data Source
AI summary
One example method includes defining a time window during which a drift mode of a machine learning model will be evaluated, and the time window begins at t0, training, beginning at the time t0, a reduced reference model R0 with a data sample d0 that is a subset of a training dataset that was used to train the machine learning model; for ‘n’ iterations: at a time ti, when new data samples di are available, train a new version Ri of the reduced reference model R0; after Ri is trained, collect data vi; and compare respective inference values generated by the reduced reference model R0 and Ri, using the data samples d0 and vi, and store an absolute difference between the inference values; defining a drift mode curve using the absolute differences; and based on the drift mode curve, determining a drift mode of the machine learning model.