Model Explanation via Feature Contribution Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As machine learning systems become increasingly complex, it becomes difficult to understand why a model generates a given score or makes a decision, especially when model inputs are outputs of other models or undergo transformations, leading to a need for explaining machine learning models and their decisions.
Innovation Solution
The development of systems and methods that use differential credit assignment methods like Aumann-Shapley and measure-theoretic extensions to quantify the importance of each input variable in model-based decisions, allowing for the explanation of how changes in input values affect model outputs and enabling the identification of contributions from individual features or groups of features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models become more complex to improve predictive accuracy, then model performance is improved, but model interpretability deteriorates
Solution Approach 1:
The patent segments the model's predictive output into individual feature contributions by computing integrated gradients for each input variable. This decomposition allows the complex model to be analyzed in terms of simpler, interpretable components, resolving the contradiction between model complexity and interpretability.
Solution Approach 2:
The patent introduces an intermediary explanation layer that translates complex model predictions into human-readable feature importance statements. This intermediary layer bridges the gap between the complex model's internal reasoning and human interpretability requirements.
2Reliability
If model inputs are transformed through multiple models and processing stages, then predictive capability is improved, but explanation difficulty increases
Solution Approach 1:
The patent implements a feedback mechanism that traces predictions back through transformation stages to identify which input features drove the final output. This feedback loop enables explanations to be generated despite multiple processing stages, reducing explanation difficulty while maintaining predictive capability.
Solution Approach 2:
The patent performs preliminary computation of feature importance weights during the transformation process itself, rather than attempting to explain after the fact. This preliminary action simplifies the explanation generation step by pre-computing the necessary attribution information through the transformation pipeline.
Data Source
AI summary
Methods, non-transitory computer readable media, and model evaluations systems for understanding diverse machine learning models (MLMs) are disclosed. In some examples, a feature contribution value is determined for features included in a reference or evaluation input data set. The evaluation input data set represents a protected class population and each feature contribution value identifies a contribution by a feature to a difference in output generated by an MLM for the evaluation input data set. Model explanation information is generated using the feature contribution values and execution of the MLM is monitored. The model explanation information explains the difference in output generated by the MLM for the evaluation input data set and includes information relating to a model-based decision. A report is generated from a knowledge graph for the MLM and output via a GUI to an operator device that includes the model explanation information.