Neural Network Explainability via Relevance Vector Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, despite their accuracy, are challenging to use in applications requiring accountability and transparency due to their 'black box' nature, where learned patterns and embeddings are hidden in neuron weights and biases, making it difficult to retrieve the factors leading to predictions without complex analyses.
Innovation Solution
The implementation of an explainable AI (XAI) system that generates global explainability datasets using relevance vectors computed from neural network weights and biases, allowing for the quantification of input feature importance and providing insights through visualizations and interpretations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning models are used to improve prediction accuracy, then prediction accuracy is improved, but transparency and accountability deteriorate due to black box behavior
Solution Approach 1:
The patent introduces an explainable AI system as an intermediary between the deep learning model and the user. This system includes an explanation generator that processes the black box model's internal states (neuron activations, weights, biases) to produce human-interpretable explanations, thereby mediating the information loss between the accurate but opaque model and the user needing transparency
2Measurement precision
If deep learning models are used to improve prediction accuracy, then prediction accuracy is improved, but accountability deteriorates due to hidden neuron weights and biases
Solution Approach 1:
The patent implements a feedback mechanism where the explainable AI system generates explanations that can be used to verify and understand the deep learning model's decision-making process. The explanation generator analyzes the model's internal states and provides feedback in the form of interpretable explanations, allowing users to assess accountability while maintaining prediction accuracy
3Loss of information
If complex analyses are performed to retrieve factors leading to predictions, then transparency is improved, but computational complexity and cost increase
Solution Approach 1:
The patent applies preliminary action by having the explanation generator pre-process and analyze the deep learning model's internal states (weights, biases, activations) in advance to create interpretable explanations. This preliminary analysis captures the essential factors leading to predictions before they are needed for interpretation, reducing the complexity of on-demand analysis while maintaining transparency
Data Source
AI summary
The layers of a neural network model are traversed in sequence one or more times while generating a plurality of relevance scores each time based on neuron weights and neuron biases of the neuron network model. Each relevance score of the plurality of relevance scores quantifies a relevance of a neuron in a lower layer of the sequence of layers to a higher layer of the sequence of layers. One or more relevance vectors can be populated from the plurality of relevance scores generated at the one or more times. Each of the relevance scores in each relevance vector quantifies a relevance of one of the input features to a task for which the neural network model is trained to perform. An explanation of a behavior of the neural network as a whole is generated based on the one or more relevance vectors.