Stable Feature Importance Ranking via Cross-Trained Model Groups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for optimizing feature importance in binary classification models are unstable and ineffective, particularly when dealing with unbalanced data, as they often rely solely on classification error, leading to inferior stability characteristics and impractical ranking of feature importance.
Innovation Solution
The method involves generating multiple confidence threshold values, building model groups, cross-training models with different training data sets, computing classification errors, calculating feature importance values, and selecting the most stable model group based on importance stability values, which considers the variation in feature importance across models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional methods are used to measure feature importance based solely on classification error, then the measurement process is simple, but the stability and reliability of feature importance rankings deteriorate
Solution Approach 1:
The training data is divided into multiple parts to create different training data sets, and multiple binary classification models are trained on these different data sets. This segmentation allows for assessing feature importance stability across multiple model instances, resolving the contradiction by making the measurement process more reliable through multiple samples while managing complexity through systematic organization.
Solution Approach 2:
Multiple binary classification models are trained in advance on different training data sets before the actual feature importance measurement. This preliminary action creates a ensemble of models whose feature importance rankings can be compared to assess stability, thereby improving reliability while the complexity is managed through the structured multi-step process.
2Reliability
If multiple models are trained to assess feature importance stability, then the stability of feature importance ranking improves, but the training time and computational resources increase
Solution Approach 1:
The training data is segmented into multiple parts to create different training data sets, enabling parallel training of multiple models. This segmentation allows for efficient utilization of computational resources and reduces sequential training time while maintaining the reliability benefits of multiple model assessments.
Solution Approach 2:
Instead of training an excessive number of models, the method uses a sufficient number of models (multiple but not excessive) to achieve stable feature importance rankings. This partial action approach balances the trade-off between reliability improvement and time consumption by using just enough models to achieve stability without unnecessary computational overhead.
3Measurement precision
If feature importance is measured using conventional methods, then the process is fast, but the accuracy and robustness of feature selection deteriorates with unbalanced data
Solution Approach 1:
Multiple binary classification models are trained in advance on different training data sets before the actual feature importance measurement. This preliminary action creates a robust foundation for accurate feature importance assessment by capturing variations in model behavior, thereby improving measurement precision while the structured process manages the time investment efficiently.
Solution Approach 2:
Multiple copies (instances) of binary classification models are created and trained on different training data sets. By comparing feature importance rankings across these model copies, the method achieves more accurate and robust feature importance measurement that is less sensitive to data imbalances, while the systematic approach maintains reasonable processing efficiency.
Data Source
AI summary
Feature importance is critical to understanding how predictive models produce accurate results, and can change significantly for different models. The present invention is used to achieve a good ranking for stable feature importance. An optimized technique is presented which considers feature importance value variation within different groups of cross-trained models. Feature importance is computed for all group models with this optimized method, and then a best set of models can be selected based on classification error as well as optimized stable feature importance values.