GAMI-Tree Training for Interpretable Feature Interaction Modeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models struggle to accurately identify feature interactions, leading to inaccurate model interpretability and performance, particularly in high-dimensional datasets.
Innovation Solution
The GAMI-Tree model uses model-based trees as base learners for main and interaction effects, employs an iterative training method, and incorporates a new interaction filtering method to capture feature interactions more accurately, resulting in improved model performance and interpretability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complex machine learning models are used to capture high-dimensional patterns, then predictive performance is improved, but model interpretability deteriorates
Solution Approach 1:
The patent segments the complex model into two distinct components: a complex base model (e.g., neural network, gradient boosting) for capturing high-dimensional patterns, and a simplified surrogate model (e.g., linear model, decision tree) for providing interpretable explanations. This segmentation allows each component to fulfill its specialized function while working together as an integrated system.
Solution Approach 2:
The surrogate model acts as an intermediary between the complex base model and the user seeking interpretation. Instead of directly interpreting the complex model's internal workings, the surrogate model mediates by learning to replicate the base model's predictions on a subset of features, thereby providing an interpretable interface to the complex system's behavior.
2Device complexity
If low-order fANOVA models are used for interpretability, then model simplicity is improved, but ability to capture high-dimensional interactions deteriorates
Solution Approach 1:
The patent performs preliminary action by training the surrogate model on residuals from the base model after main effects have been accounted for. This allows the surrogate model to focus specifically on capturing interaction effects rather than learning all patterns from scratch, improving its ability to detect interactions with limited complexity.
Solution Approach 2:
The patent transitions from analyzing individual feature effects to analyzing feature pairs by constructing interaction terms (e.g., x1*x2) as new features. This dimensional transformation allows low-order models to capture interaction effects by elevating the analysis to a higher-dimensional feature space where interactions become explicit features.
3Productivity
If gradient boosting with piecewise constant trees is used to fit GA2M models, then computational efficiency is improved, but accuracy in identifying feature interactions deteriorates
Solution Approach 1:
The patent changes the functional form parameter of the surrogate model from piecewise constant functions to more flexible functions such as linear models, polynomial models, or smooth splines. This parameter change allows the model to capture interaction patterns with greater accuracy while maintaining computational efficiency through the use of efficient optimization algorithms.
Data Source
AI summary
Systems, apparatuses, methods, and computer program products are disclosed for training a GAMI-Tree model. An example method includes initializing an iterative prediction model and performing a required number of model training iterations. For each model training iteration, the method further includes (i) performing a required number of main-effect gradient boosting iterations of a main-effect gradient boosting routine, (ii) generating a plurality of qualified input feature pairs, and (iii) performing a required number of interaction-effect gradient boosting iterations of a interaction-effect gradient boosting routine. The method further includes generating the GAMI-Tree model based on the iterative prediction model generated by a final interaction-effect gradient boosting iteration of a final model training iteration.