Learning Model Management System for Prediction Accuracy Deterioration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In systems using machine learning, the prediction accuracy of learning models deteriorates over time due to insufficient evaluation and update of learning data, particularly when the frequency of new data input is low, leading to increased operating costs and reduced reliability in maintaining necessary prediction accuracy.
Innovation Solution
A learning model management system that performs provisional evaluations using a smaller amount of feedback data and provides notifications for performance deterioration, allowing for timely updates and relearning without continuous active data collection, thereby maintaining prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If evaluation is performed using a small amount of learning data, then operating costs are reduced, but the reliability of evaluation results deteriorates
Solution Approach 1:
The system performs preliminary actions by storing multiple candidate learning models before formal evaluation. When data is limited, it evaluates these pre-stored models rather than requiring extensive new data collection, thus reducing operating costs while maintaining evaluation reliability through multiple candidates.
Solution Approach 2:
The system introduces an intermediary mechanism - a storage unit that holds multiple candidate learning models generated in advance. This intermediary allows evaluation to proceed with limited data by selecting from pre-prepared models, bridging the gap between cost reduction and reliability maintenance.
2Reliability
If continuous learning cycle is executed at high frequency to maintain prediction accuracy, then prediction accuracy is maintained, but operating costs increase
Solution Approach 1:
The system implements periodic action by storing multiple candidate learning models in advance and evaluating them at scheduled intervals rather than continuously. This allows maintaining prediction accuracy through regular updates while reducing operating costs by avoiding constant data collection and model retraining.
Solution Approach 2:
Multiple candidate learning models are prepared in advance through preliminary learning processes. This allows the system to perform evaluations by selecting from pre-prepared models rather than continuously collecting and processing new data, reducing operating costs while maintaining the ability to update prediction accuracy periodically.
3Reliability
If learning data is actively collected at all times, then evaluation reliability is improved, but system resource costs increase
Solution Approach 1:
The system performs preliminary actions by storing multiple candidate learning models in advance. This eliminates the need for continuous data collection during evaluation, as the models are already prepared. Evaluation reliability is maintained through multiple candidates while system resource costs are reduced by avoiding continuous active data collection.
Solution Approach 2:
The storage unit containing multiple candidate learning models acts as an intermediary that decouples evaluation from continuous data collection. This allows reliable evaluation to proceed using pre-stored models while avoiding the high system resource costs associated with continuous active data collection.
4Loss of energy
If evaluation is not performed when learning data is insufficient, then operating costs are reduced, but prediction accuracy deterioration goes undetected
Solution Approach 1:
The system performs preliminary actions by storing multiple candidate learning models in advance. When data is insufficient for new model training, it can still evaluate existing candidates from storage, enabling detection of performance deterioration without incurring the costs of active data collection and model retraining.
Solution Approach 2:
The storage unit of candidate models serves as an intermediary that enables evaluation to proceed even when learning data is insufficient. This allows the system to detect performance deterioration through evaluation of pre-stored models while maintaining reduced operating costs by avoiding active data collection.
Data Source
AI summary
A learning model management system executes a provisional evaluation when the number of feedback data is equal to or less than a threshold of a definite evaluation but is more than a threshold of a provisional evaluation. In this provisional evaluation, prediction accuracy of the learning model that is in operation is evaluated a plurality of times, and whether or not the prediction accuracy of the learning model that is in operation is in a deterioration trend is determined based on the change tendency of the evaluation results. If it is determined that the prediction accuracy of the learning model that is in operation is in a deterioration trend, the learning model management system provides a notification about the deterioration trend of the prediction accuracy to a manager to cause him/her to increase the number of feedback data up to a number that enables the definite evaluation.