ML Model Retraining Management via Drift Permanence Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing method for retraining machine learning models does not distinguish between the causes of concept drift, leading to unnecessary retraining, reduced accuracy, and improper timing, resulting in increased costs and decreased model performance.
Innovation Solution
A management apparatus with a processor and storage device that generates a machine learning model, performs inference processes, and determines the necessity of retraining based on operation influence information, management operation logs, and schedules, distinguishing between temporary and perpetual differences in actual and predicted monitoring data to decide on retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the ML model is retrained whenever concept drift is detected, then the model accuracy can be maintained, but unnecessary retraining increases cost and reduces efficiency
Solution Approach 1:
The patent applies local quality by analyzing the specific characteristics of concept drift to determine whether retraining is necessary. Instead of uniform retraining, the system evaluates the local nature of the drift (e.g., whether it's caused by one-time operations vs. continuous changes) and applies retraining only when appropriate, thus optimizing the balance between maintaining accuracy and avoiding unnecessary retraining costs
2Adaptability or versatility
If the ML model is retrained frequently to adapt to changes, then the model can handle concept drift, but retraining timing becomes improper and extra cost is incurred
Solution Approach 1:
The patent applies preliminary action by detecting concept drift early and analyzing its characteristics before initiating retraining. The system evaluates whether the detected drift requires immediate retraining or can be addressed later, allowing for optimal timing that balances adaptability with cost efficiency. This prevents premature or unnecessary retraining while ensuring timely response to genuine model degradation
3Reliability
If retraining is performed to address concept drift, then model performance can be improved, but the complexity of managing retraining processes increases
Solution Approach 1:
The patent applies feedback by implementing a systematic evaluation process that continuously monitors model performance and concept drift characteristics. The system uses feedback from drift analysis to automatically determine whether retraining is necessary, reducing the complexity of manual management. The feedback loop includes detecting drift, analyzing its nature, and making informed retraining decisions, thereby simplifying the overall management process while maintaining high model performance
Data Source
AI summary
A management apparatus includes operation influence information defining an influence a management operation exerts on a management target, a management operation log recording a management operation executed on the management target, and a management operation schedule indicating a management operation of which execution on the management target is planned or inferred; determines whether a difference between actual measurement monitoring data acquired from the management target and predicted monitoring data indicating a result of predicting the monitoring data exceeds a given threshold; defines the difference as a significant difference when the difference exceeds the threshold; determines whether the significant difference is temporary or perpetual, based on the operation influence information, management operation log, and management operation schedule; and when determining the significant difference to be perpetual, determines that retraining of a machine learning model should be executed.