Gradient-Based Directional Drivers for Deep Learning Interpretability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, particularly those with high feature dimensionality, are difficult to interpret due to their complexity, making it challenging to quantify the impact of variables or their features on model outputs, especially in time series analysis where sequence dependence complicates the explanation of model outputs.
Innovation Solution
The system addresses this by grouping features into variables and variables into groups, using gradients to aggregate impacts, and collapsing temporal dimensions to improve explainability, thereby identifying directional drivers that indicate the impact of feature groups on model outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep learning models use large numbers of features for training and execution, then model sophistication and complexity increase, but output interpretability deteriorates
Solution Approach 1:
The patent segments the high-dimensional feature space by grouping features into variables and variables into variable groups based on domain knowledge. This hierarchical segmentation transforms the black-box model into an interpretable structure where features are organized into meaningful categories, allowing users to understand model outputs at multiple levels of aggregation without sacrificing model sophistication.
2Adaptability or versatility
If feature dimensionality is increased, then model capability improves, but output manageability deteriorates
Solution Approach 1:
The patent introduces a new dimensional organization by creating hierarchical groupings (features → variables → variable groups) that add semantic dimensions to the feature space. This dimensional transformation allows the model to maintain high feature dimensionality for capability while improving manageability through structured organization that maps to domain concepts.
3Quantity of substance
If features are derived from the same variable, then feature richness improves, but variable impact quantification deteriorates
Solution Approach 1:
The patent merges gradients from multiple features derived from the same variable by aggregating them at the variable level. This combining approach allows the model to retain rich feature representations while producing unified variable-level impact measurements, resolving the contradiction between feature richness and precise variable impact quantification.
4Measurement precision
If sequence dependence among input variables is considered, then model accuracy improves, but variable impact explanation deteriorates
Solution Approach 1:
The patent performs preliminary aggregation of feature gradients into variable gradients and variable group gradients before final interpretation. This preliminary action captures sequence dependence effects in the gradient computations while organizing results hierarchically, preserving explanatory information about variable impacts even when sequence dependence is considered in the model.
Data Source
AI summary
The disclosure relates to systems and methods of determining gradient-based directional drivers of deep learning models. A system may access a plurality of features and a group definition that specifies one or more groups of features. The system may provide the plurality of features as input to a deep learning model trained to generate a model output based on a model function and the plurality of features. The system may obtain, for each feature, a gradient that represents a rate of change of the model function based on the feature and then aggregate, based on the group definition, the gradients obtained from the deep learning model; and for each group of features from among the one or more groups of features: determine a directional driver based on the aggregated gradients, the directional driver indicating an impact of the group of features on the model output.