Local Explainability Dataset for Neural Network Prediction Transparency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models lack transparency and accountability, making it difficult to understand the factors leading to their predictions, which limits their usability in applications requiring explainability.
Innovation Solution
A method and system for generating local explainability of neural network predictions by extracting feature relevance information from neuron weights and biases, using an XAI model to create a local explainability dataset that can be transformed into user-friendly formats like graphs and visual indicators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning models are used to improve prediction accuracy, then prediction accuracy is improved, but transparency and accountability deteriorate due to black box behavior
Solution Approach 1:
The patent introduces an XAI model as an intermediary between the deep learning model and users. This XAI model receives the test input vector and generates a local explainability dataset that reveals which input features most influenced the prediction, thereby mediating the information loss between the black box model and the need for transparency
Solution Approach 2:
The patent segments the explanation generation process into distinct components: (1) feeding the test input vector to the deep learning model, (2) generating relevance scores for each input feature, and (3) creating a local explainability dataset. This segmentation allows the system to maintain prediction accuracy while providing structured, interpretable explanations
2Measurement precision
If deep learning models continuously learn from new transactions to improve accuracy, then prediction accuracy is improved, but complexity of analysis increases to retrieve prediction factors
Solution Approach 1:
The patent implements self-service by having the deep learning model generate its own explanation data automatically. The model produces relevance scores and local explainability datasets without requiring external complex analysis tools, allowing the system to maintain accuracy while reducing the complexity of retrieving prediction factors
3Ease of operation
If deep learning models are deployed in applications requiring accountability, then usability is improved, but the black box behavior limits usability
Solution Approach 1:
The patent implements feedback by generating local explainability datasets that provide actionable insights about prediction factors. This feedback loop allows users to understand why specific predictions were made, enabling them to take appropriate actions while maintaining accountability in applications requiring it
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A network output is generated by feeding an input vector to an input layer of a neural network model having a plurality of neurons arranged in a sequence of layers, a plurality of neuron weights, and a plurality of neuron biases. The network output is used to determine an output relevance score. Relevance scores at a last layer of the sequence of layers are generated. Relevance scores are obtained at a first layer of the sequence of layers by reverse propagating the relevance scores generated at the last layer through the sequence of layers other than the last layer using the neuron weights and neuron biases. A feature relevance vector is formed based on the input vector and the relevance scores obtained at the first layer and included in a local explainability dataset, which is then used to generate a local explanation of a prediction of the neural network model.