Neural Network Explainability via Intermediary Logistic Regression Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models lack explainability, making it difficult to grasp the importance of features contributing to predictions, and existing methods like linear regression are inefficient or contradictory in achieving this explainability.
Innovation Solution
A data analysis apparatus with a neural network configuration that includes an input layer, intermediate layers, and an output layer, featuring a conversion section, reallocation section, and importance calculation section, which converts and reallocates data to facilitate the calculation of feature importance using learning parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning models are used to achieve high prediction accuracy, then prediction accuracy is improved, but explainability of feature importance deteriorates
Solution Approach 1:
The patent introduces an intermediary explanation model (logistic regression) that mediates between the deep learning model's complex nonlinear decisions and the user's need for interpretable feature importance. The explanation model learns to replicate the deep learning model's predictions while maintaining linear relationships that allow for clear feature importance calculation through weights, thus preserving accuracy while restoring explainability.
Solution Approach 2:
The patent segments the explanation task from the prediction task by using two separate models: the deep learning model for accurate prediction and the explanation model for interpretable feature importance. This segmentation allows each model to optimize for its specific function without compromising the other, enabling high accuracy predictions while providing clear explanations through the segmented explanation pathway.
2Loss of information
If linear regression is used to calculate feature importance for deep learning predictions, then explainability is improved, but calculation completeness and accuracy deteriorate
Solution Approach 1:
The explanation model acts as an intermediary that learns the mapping from input features to predictions made by the deep learning model. By training this intermediary model to replicate the deep learning model's output, the system captures the complete decision logic while maintaining linear relationships that enable accurate and complete feature importance calculation through the model's weights.
Solution Approach 2:
The patent changes the parameter representation from the deep learning model's distributed nonlinear weights to the explanation model's linear weights. This parameter transformation allows the same predictive function to be represented in a form where feature importance is directly observable and complete, as each weight corresponds to a specific feature's contribution to the prediction.
3Productivity
If logistic regression is used to calculate feature importance, then calculation efficiency is improved, but individual sample importance calculation capability deteriorates
Solution Approach 1:
The explanation model is designed to be self-sufficient in calculating feature importance for any input sample. By training the model to learn the relationship between inputs and predictions during the training phase, the model automatically acquires the capability to calculate feature importance for individual samples during inference, eliminating the need for additional processing or separate analysis steps.
Data Source
AI summary
Facilitation of an explanation about an object to be analyzed is realized with high accuracy and with efficiency.A data analysis apparatus is disclosed which uses a first neural network configured with an input layer, an output layer, and two or more intermediate layers provided between the input layer and the output layer. Each performs a calculation by giving data from a layer of a previous stage and a first learning parameter to a first activation function and outputs a calculation result to a layer of a subsequent stage. The data analysis apparatus includes a conversion section; a reallocation section; and an importance calculation section.