Functional Activation Analysis for Deep Neural Network Explainability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for explainable artificial intelligence (AI) face challenges such as overfitting, reduced generalizability, and scalability issues when analyzing deep neural networks.
Innovation Solution
The method involves accessing input data and a pretrained neural network, forming a block-sequence to apply the input data, generating a time-series of layer output values, and processing these values to generate neural network activation data, which indicate the degree of activation of each layer in response to the input data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If localization techniques are used to identify specific components within a neural network, then explainability is improved, but model performance and generalizability deteriorate due to overfitting
Solution Approach 1:
The patent introduces functional networks as an intermediary layer between the neural network components and the explanation output. These functional networks aggregate activations from multiple neurons and layers, providing a higher-level abstraction that explains model behavior without requiring precise localization of individual components, thereby avoiding overfitting while maintaining explainability
Solution Approach 2:
The patent transitions from analyzing individual neurons or layers to analyzing functional networks that span multiple dimensions of the neural network. By creating a new dimensional space of functional network activations, the method provides explainability without being constrained by the high-dimensional complexity of individual component localization
2Loss of information
If localization techniques are used to identify specific components within a neural network, then explainability is improved, but scalability deteriorates as network size increases
Solution Approach 1:
The patent creates a universal functional network framework that can be applied to neural networks of any size or architecture. The functional networks serve multiple purposes: they aggregate information from different layers, provide explainability, and can be computed efficiently regardless of the underlying network complexity, making the approach scalable to large models
3Measurement precision
If sparse autoencoder approach is used to extract features, then feature extraction capability is improved, but alignment with human understanding deteriorates
Solution Approach 1:
The patent changes the parameters used to define functional networks from purely data-driven sparse features to parameters that incorporate human-understandable concepts. By adjusting the functional network definitions to align with human categories and interpretations, the method maintains feature extraction capability while improving alignment with human understanding
4Measurement precision
If sparse autoencoder approach is used to extract features, then feature extraction capability is improved, but scalability deteriorates due to increasing autoencoder size
Solution Approach 1:
The patent extracts only the essential functional network activations needed for explanation, rather than maintaining a comprehensive autoencoder that scales with the entire neural network. By taking out and analyzing only the relevant functional components, the method provides effective feature extraction without requiring the autoencoder size to increase proportionally with network complexity
Data Source
AI summary
Functional activation-based analysis of deep neural networks uses a structured set of inputs (e.g., input datasets corresponding to different knowledge or datatype domains) are sequentially provided to a pretrained neural network (e.g., according to a block-sequence). The output values for each node in the neural network are recorded and stored as a time-series of layer output values. A statistical analysis of the time-series of layer output values may be fit as a function of the structured set of inputs to generate neural network analysis data that indicate activations of layers within the neural network based on the inputs.