Tree Model Feature Importance via Node Gain Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for computing feature importance in tree-based models fail to capture two-or-more-way interactions between features and introduce inherent bias, limiting their effectiveness for model pruning and feature selection.
Innovation Solution
A method that computes variable relevance by considering the entire tree structure, calculating gain contribution values and updating node and variable gain values to select relevant variables, thereby capturing n-way feature interactions and avoiding bias.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If split-based node contribution methods are used to compute feature importance, then the computation is simple and fast, but the method cannot capture two-or-more-way interactions between features and introduces inherent bias
Solution Approach 1:
The patent segments the tree structure into individual nodes and computes contribution at each node level, then aggregates these contributions to capture global feature importance while maintaining computational efficiency. This segmentation allows the method to consider the entire tree structure rather than just local splits.
Solution Approach 2:
The patent merges the advantages of split-based methods (computational efficiency) with tree-based methods (capturing interactions) by combining local node contribution calculations with global tree structure consideration, achieving both speed and accuracy.
2Measurement precision
If resampling feature importance methods are used to compute feature importance, then the method can capture feature interactions, but the method does not yield a node contribution value and only provides a single output score per feature
Solution Approach 1:
The patent segments the feature importance computation into node-level contributions that can be aggregated, preserving detailed information about which parts of the tree contribute to feature importance while still capturing global interactions.
Solution Approach 2:
The patent uses a feedback mechanism where node contributions are computed based on impurity reduction and population ratios, then aggregated to provide both local and global feature importance measures, ensuring no information is lost in the process.
3Ease of operation
If the same heuristic is used for training the tree and evaluating feature importance, then the process is consistent and simple, but inherent bias is introduced
Solution Approach 1:
The patent segments the evaluation process to compute contributions at each node independently, then aggregates these contributions. This segmentation allows the method to maintain consistency in the training process while reducing bias through the aggregation of multiple local decisions.
Solution Approach 2:
The patent changes the parameter used for evaluation from the training heuristic to a contribution-based metric that considers population ratios and impurity reduction, thereby reducing bias while maintaining process consistency.
Data Source
AI summary
A computing system computes a variable relevance using a trained tree model. (A) A next child node is selected. (B) A number of observations associated with the next child node is computed. (C) A population ratio value is computed. (D) A next leaf node is selected. (E) First observations are identified. (F) A first impurity value is computed for the first observations. (G) Second observations are identified when the first observations are associated with the descending child nodes. (H) A second impurity value is computed for the second observations. (I) A gain contribution is computed. (J) A node gain value is updated. (K) (D) through (J) are repeated. (L) A variable gain value is updated for a variable associated with the split test. (M) (A) through (L) are repeated. (N) A set of relevant variables is selected based on the variable gain value.