Relevance Score Assignment for Neural Network Interpretability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks lack interpretability, making it difficult for human experts to verify classification decisions, especially in applications where detailed information about the classification process is necessary, as they operate as 'black boxes' and do not provide explanations for their predictions.
Innovation Solution
A method for assigning relevance scores to items by reversing propagating an initial relevance score through the neural network, redistributing it among neurons using a distribution function, allowing for the identification of key contributions to the classification decision, such as specific pixels in image classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex machine learning algorithms are used to improve classification performance, then classification accuracy is improved, but interpretability deteriorates
Solution Approach 1:
The patent segments the black-box classification process into interpretable components by introducing relevance scores that decompose the decision-making process. The relevance propagation mechanism breaks down the complex neural network computation into traceable contributions from individual input features, allowing stakeholders to understand which features drove the classification decision without sacrificing accuracy.
Solution Approach 2:
The patent introduces relevance scores as an intermediary layer between the input features and the classification output. These relevance scores act as mediators that translate the internal complex computations of the neural network into human-understandable explanations, bridging the gap between accurate black-box predictions and interpretable decision reasoning.
2Measurement precision
If neural networks with multiple nonlinear mappings are used to improve classification performance, then classification accuracy is improved, but interpretability deteriorates
Solution Approach 1:
The patent inverts the traditional forward propagation approach by implementing backward relevance propagation. Instead of only computing predictions forward through the network, the system propagates relevance information backward from the output to the input, enabling interpretation of the complex nonlinear mappings without requiring simplification of the network architecture itself.
Solution Approach 2:
The patent adds a new dimension to the neural network computation by introducing the relevance score dimension. This additional dimension transforms the single-output prediction problem into a multi-dimensional problem that includes both the classification output and the interpretability information, allowing the network to maintain its complex architecture while providing explainable results.
3Difficulty of detecting and measuring
If detailed explanation of classification decisions is provided to improve interpretability, then interpretability is improved, but computational complexity increases
Solution Approach 1:
The patent implements partial action by computing relevance scores only when interpretability is needed, rather than continuously during normal classification operations. The system can operate in a computationally efficient mode for pure prediction and switch to the relevance propagation mode only when explanations are required, thus providing detailed interpretability information without permanently increasing computational overhead.
Data Source
AI summary
The task of relevance score assignment to a set of items onto which an artificial neural network is applied is obtained by redistributing an initial relevance score derived from the network output, onto the set of items by reversely propagating the initial relevance score through the artificial neural network so as to obtain a relevance score for each item. In particular, this reverse propagation is applicable to a broader set of artificial neural networks and/or at lower computational efforts by performing same in a manner so that for each neuron, preliminarily redistributed relevance scores of a set of downstream neighbor neurons of the respective neuron are distributed on a set of upstream neighbor neurons of the respective neuron according to a distribution function.