Machine Learning Model Degradation Identification via Feature Impact Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Identifying performance degradation in machine learning models, particularly when dataset values are derived from a combination of outputs from different models, is challenging due to the obscurity of the process, making it difficult to determine which model's performance has degraded.
Innovation Solution
A system processes a hierarchy of features to identify the target feature contributing to the difference between actual and predicted results, calculating impact values at each level to pinpoint the degraded model, allowing for retraining to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple machine learning models are used to generate predicted values for dataset features, then the accuracy and comprehensiveness of predictions are improved, but it becomes difficult to identify which specific model's performance has degraded
Solution Approach 1:
The patent segments the overall prediction accuracy into individual model contributions by calculating impact values for each model's predicted features. This allows identification of which specific model is causing performance degradation while maintaining the benefits of using multiple models collectively.
Solution Approach 2:
The patent introduces impact values as an intermediary metric that bridges the gap between overall prediction accuracy and individual model performance. These impact values serve as a mediator to trace and identify the specific model responsible for performance degradation without requiring direct observation of each model's output.
2Difficulty of detecting and measuring
If the system traces features through a hierarchy to identify the source of performance degradation, then the ability to pinpoint the degraded model is improved, but the computational complexity and processing time increase
Solution Approach 1:
The patent performs preliminary calculations of impact values for all features and models before conducting the full hierarchy traversal. This pre-computation allows the system to quickly identify promising paths in the feature hierarchy without examining every possible feature combination, thereby reducing processing time while maintaining identification accuracy.
Data Source
AI summary
Methods and systems are described herein for identifying performance degradation in machine learning models based on comparisons of actual and predicted results. The system may receive predicted and actual results datasets for features within a system with the predicted results being generated by a machine learning model corresponding to the feature. The system may access a hierarchy associated with the features and select a level of the hierarchy having a subset of features. The subset of features may be associated with a target feature. Impact values may then be generated for the subset, where the impact values indicate contributions of the corresponding features to a difference, in the target feature, between predicted and actual results. The system may select a new target feature, from the subset, associated with a highest impact value and may retrain a machine learning model associated with the new target feature.