Variable Importance Assessment via Input Shuffling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing predictive models lack transparency in determining how individual input variables influence outcomes, especially in complex models, and current methods do not provide a clear measure of variable impact on a case-level basis, making it difficult to interpret results and troubleshoot inaccurate predictions.
Innovation Solution
A method that involves randomly resampling values for each input variable while keeping other variables constant, using bootstrap sampling, to generate new scores and calculate the standard deviation between original and resampled scores, thereby determining the impact of each variable on the predictive model's outcomes, and identifying variables with significant impact by exceeding a specified threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex predictive modeling techniques are used to improve prediction accuracy, then the accuracy of outcomes is improved, but the transparency of the model in determining how variables influence outcomes deteriorates
Solution Approach 1:
The patent segments the overall model evaluation into individual variable-level assessments. By evaluating each variable's impact separately through random resampling and score comparison, the method breaks down the complex model's decision-making process into manageable, interpretable components that can be analyzed independently while maintaining overall model accuracy.
Solution Approach 2:
The patent introduces an intermediary assessment layer between the complex model and the user. This layer uses random resampling and standard deviation calculation to create intermediate metrics (score variations) that bridge the gap between the opaque complex model and human interpreters, making the model's variable dependencies visible without simplifying the underlying complex algorithms.
2Measurement precision
If complex predictive models are used to achieve more accurate outcomes, then the accuracy is improved, but the ease of understanding variable influence deteriorates
Solution Approach 1:
The patent enables the model to self-explain its own behavior through automated variable importance assessment. The system performs self-service by automatically conducting random resampling, calculating score deviations, and generating variable importance rankings without requiring external analysis tools or expert intervention, making complex model interpretation accessible to end users.
Solution Approach 2:
The patent changes the parameter of variable values through random resampling to observe model response. By systematically varying individual variable values while holding others constant and measuring the resulting score changes, the method transforms the unobservable internal variable relationships into measurable parameters that indicate variable importance and direction of influence.
3Loss of information
If case-level variable importance assessment is implemented, then the transparency and interpretability are improved, but the computational complexity and time required deteriorates
Solution Approach 1:
The patent applies partial action by focusing computational effort only on individual variable resampling rather than complete model retraining. By randomly resampling single variable values and assessing their impact in isolation, the method achieves sufficient interpretability information with fewer computational iterations than would be required for comprehensive model analysis, balancing detail with efficiency.
Data Source
AI summary
Techniques are disclosed herein for determining variable importance on a predictive model on a case level. Modeling data associated with a case is received. The modeling data provides input variables, each having a corresponding value for input to a predictive modeling technique associated with the case. A measure of impact for each of the variables is determined using an input shuffling method. Variables having a measure of impact that exceeds a specified threshold are identified. A summary that includes the identified variables is generated.