ML Framework for Feature Importance via Null Hypothesis Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current feature importance methods in machine learning, such as permutation feature importance (PFI), provide relative but not absolute importance scores, lack statistical inference, and are inconsistent and unstable due to reliance on models like Gradient Boosting Machines (GBM) that may not capture complex relationships between features and user traffic.
Innovation Solution
A framework that incorporates Null Hypothesis Significance Testing (NHST) and injects artificial noisy features into machine-learned model generation, allowing for statistical correlation testing between response variables and event categories, thereby improving the consistency and stability of feature importance assessments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If permutation feature importance (PFI) is used to assess feature importance, then relative importance scores can be obtained, but absolute importance scores and statistical inference are not provided
Solution Approach 1:
The patent introduces Null Hypothesis Significance Testing (NHST) as an intermediary statistical framework between feature importance assessment and decision-making. NHST provides a mediator that enables statistical inference by testing whether observed feature importance scores differ significantly from what would be expected by chance, thereby bridging the gap between relative scoring and absolute importance determination.
Solution Approach 2:
The patent changes the parameter being measured from relative importance scores to absolute importance scores with statistical significance. By transforming the measurement approach to include p-values and significance testing, the system transitions from providing only ranked relative importance to providing statistically inferred absolute importance levels.
2Productivity
If Gradient Boosting Machines (GBM) are used for model generation, then machine learning predictions can be made, but feature importance assessments become inconsistent and unstable
Solution Approach 1:
The patent implements a feedback mechanism through NHST that continuously evaluates and validates feature importance assessments. By testing statistical significance and comparing results against null hypotheses, the system provides feedback that identifies and eliminates unstable or unreliable importance rankings, thereby improving consistency without sacrificing model generation efficiency.
3Adaptability or versatility
If multiple machine-learned models are generated based on different sets of training data, then comprehensive predictions can be made, but computational resources and time are consumed
Solution Approach 1:
The patent segments the model generation process by creating multiple specialized models, each trained on specific subsets of training data tailored to different event categories or user segments. This segmentation allows the system to maintain high adaptability to different data types while reducing overall computational time by focusing each model on specific tasks rather than training comprehensive models on all data.
Data Source
AI summary
A framework for generating multiple machine-learned (ML) models in order to learn the impact of real-world events is provided. In one technique, sets of feature values (FVs) are stored, each FV set corresponding to a different feature. Also, sets of artificial FVs (AFVs) are generated. For each generated AFV set: (1) a training dataset is generated based on that AFV set and the multiple FV sets; (2) a model is generated based on the training dataset; (3) a ranking of the multiple features is generated based on the model; and (4) the ranking is stored in a dictionary corresponding to the generated AFV set. For each feature, a rank pair of the feature and an AF is determined from each dictionary. Based on a set of rank pairs associated with the feature, it is determined whether there is significant correlation between the feature and a response variable of the models.