Neural Network Concept Testing via Backpropagated CAV Attribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current XAI approaches for neural networks are limited by their focus on single neurons or channels, restricting concept space and failing to consider overlaps or human-defined concepts, making it difficult to interpret and verify model decisions, especially in safety-critical applications.
Innovation Solution
The method involves obtaining a concept activation vector (CAV) to process attributions through the neural network using backpropagation, allowing for the detection and localization of concepts across multiple channels, thereby enhancing interpretability and enabling the testing of predefined concepts within the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If local attribution methods are used to provide transparency for single model decisions, then interpretability for single inputs is improved, but the concept space is restricted to single neurons or channels
Solution Approach 1:
The patent combines multiple neurons or channels into a single concept vector that represents a unified concept. Instead of analyzing individual neurons separately, the method merges their contributions into a cohesive concept representation that can be attributed to model decisions, thereby expanding the concept space while maintaining interpretability.
Solution Approach 2:
The concept vector serves multiple functions: it represents a semantic concept, aggregates information from multiple neurons/channels, and enables attribution analysis. This multi-functionality allows the same mechanism to handle both local interpretability and broader concept space coverage.
2Measurement precision
If global explainability methods are used to examine model behavior generally, then broader model understanding is improved, but overlap of different concepts in latent space is not considered
Solution Approach 1:
The patent applies local quality by allowing different concept vectors to have distinct characteristics and representations in the latent space. Each concept is treated with its own unique vector representation that captures its specific properties, enabling the model to handle overlapping concepts by differentiating them through their local qualities rather than forcing a single global representation.
3Ease of operation
If concepts are split into subspaces or channels for analysis, then local attribution is simplified, but semantic overlap between concepts is lost
Solution Approach 1:
The concept vector acts as a composite representation that integrates information from multiple neurons and channels while preserving semantic relationships. Rather than treating concept components as separate immutable entities, the composite vector allows information to be combined in a way that maintains semantic overlap and relationships between concepts.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present disclosure relate to an apparatus (600), a computer program, and a method (100) for testing a neural network for a concept. The method provides for passing an output of the neural network backwards through the neural network and, in doing so, processing one or more attributions of the concept layer based on the CAV to obtain a projected attribution. As well, the method suggests testing the neural network for the concept using the projected attribution.