Time-Aware ML Explainability for Sequential Feature Attribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional feature attribution methods for black box machine learning models, such as Shapley measures, fail to account for the timing of features within a sequence, leading to misleading explainability measures and decreased model performance due to the inability to detect temporal anomalies.
Innovation Solution
Implement an explainability pipeline (ORD-SHAP) that adapts traditional explainability functions to feature attributions on sequential models, using a weighted least squares approach to quantify the effect of feature ordering and identify interaction effects between features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional feature attribution methods (e.g., Shapley measures) are used for black box machine learning models, then feature importance can be calculated, but timing information of features within sequences is lost leading to misleading explainability
Solution Approach 1:
The patent introduces a temporal dimension to traditional feature attribution by creating time-aware Shapley values. Instead of treating features as a static set, the method orders features temporally and calculates attribution values that reflect both feature importance and timing. This transforms the explanation from a single-dimension (feature importance) to multi-dimension (feature importance + temporal position), preventing information loss about when features occur in sequences.
Solution Approach 2:
The patent segments the feature attribution process into temporal intervals or time steps. By dividing the sequence into discrete time-based units and calculating Shapley values for each segment, the method preserves timing information while maintaining computational feasibility. This segmentation allows the explainability method to capture when specific features contribute to predictions without treating the entire sequence as a monolithic unit.
2Device complexity
If feature ordering is fixed in attribution methods, then computation is simplified, but the conflation of feature value attribution and position attribution leads to misleading explanations
Solution Approach 1:
The patent makes the feature ordering dynamic by considering multiple temporal orderings rather than fixing a single order. The method evaluates how feature attributions change under different temporal arrangements and synthesizes these into time-aware Shapley values. This dynamic approach allows the system to capture the relationship between feature position and importance without requiring exhaustive permutation analysis, balancing computational complexity with attribution accuracy.
Solution Approach 2:
The patent changes the parameter of feature ordering from a fixed constant to a variable that reflects temporal relationships. By introducing temporal position as a variable parameter in the Shapley value calculation, the method distinguishes between a feature's intrinsic importance and its positional contribution. This parameter change enables the system to separate value attribution from position attribution, improving measurement precision without prohibitive computational cost.
3Productivity
If temporal anomalies are not detected, then model training continues without intervention, but model performance and reliability decrease
Solution Approach 1:
The patent implements feedback by using time-aware Shapley values to monitor temporal anomalies during model training and evaluation. The explainability method provides continuous feedback about whether the model is relying on temporally inappropriate features or patterns. When temporal anomalies are detected (e.g., the model assigns high importance to features at incorrect time positions), the feedback mechanism alerts practitioners to potential reliability issues, allowing for targeted model adjustments without halting entire training pipelines.
Data Source
AI summary
Various embodiments of the present disclosure provide explainability pipelines for improving the explainability of black box machine learning model. An explainability pipeline may include generating, using a target machine learning model, a model output based on a temporally ordered input feature sequence that comprises a plurality of features respectively assigned to a plurality of time positions within the temporally ordered input feature sequence. The explainability pipeline may include generating a plurality of feature subsets and time permuted feature subsets from the temporally ordered input feature sequence. The explainability pipeline may include generating, using the target machine learning model, an evaluation output for each of the plurality of time permuted feature subsets and identifying a time impact prediction of a target feature from the plurality of features based on the evaluation outputs and the model output.