Class-Discriminative Saliency Maps for CNN Verification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for generating saliency maps in Convolutional Neural Networks (CNNs) are not class-discriminative, making it difficult to verify the internal workings and decision-making processes of CNNs, especially in security-critical applications where transparency is essential.

Innovation Solution

A method involving a contrastive layer-wise relevance propagation algorithm and a Bottom Up Attention pattern is used to generate instance-specific and class-discriminative saliency maps for each target class and feature, allowing for detailed verification of CNN classification decisions without modifying the network structure.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If traditional backpropagation-based mechanisms are used to generate saliency maps, then explanations for CNN classification decisions can be provided, but the saliency maps are not class-discriminative and recognize the same foreground objects for different target classes

Engineering Contradiction:

Improveloss of class-discriminative informationVSAvoidprecision of class-specific verification

Core Design Contradiction:

Loss of informationVSMeasurement precision

Solution Approach 1:

The patent segments the verification process into class-specific saliency map generation, where each target class receives its own dedicated saliency map through separate backpropagation operations. This segmentation allows each map to capture class-discriminative information independently, resolving the issue where traditional methods produced identical foreground object recognition across different classes.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent inverts the traditional approach by instead of generating a single aggregated saliency map, it generates multiple class-specific saliency maps through reversed backpropagation from each class output. This inversion enables the system to preserve class-discriminative information that would otherwise be lost in aggregation.

Inventive Principle:

Principle #13The other way round (Inversion)

2Measurement precision

If class-discriminative saliency maps are generated through modified CNN architecture with feedback layers, then class-specific verification is improved, but the device complexity increases due to additional layers and optimization requirements

Engineering Contradiction:

Improveprecision of class-specific verificationVSAvoidcomplexity of CNN architecture

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary verification mechanism that operates on top of the existing CNN architecture without modifying its core structure. The class-specific backpropagation acts as an intermediary process that generates verification signals using the existing network parameters, avoiding the need for additional feedback layers while still achieving class-discriminative verification.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent achieves class-discriminative verification by changing the computational parameters of the backpropagation process rather than modifying the network architecture. By adjusting the backpropagation direction and class-specific weight application, the system generates class-discriminative saliency maps using the existing CNN structure, thereby avoiding increased device complexity.

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If multiple class-specific saliency maps are generated through separate backpropagation operations, then class-discriminative verification is achieved, but the computational time and processing overhead increase

Engineering Contradiction:

Improveprecision of class-specific verificationVSAvoidcomputational time for verification

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent performs preliminary forward propagation to obtain class output scores before initiating the backpropagation process. These pre-computed scores are then reused as starting points for the class-specific backpropagation operations, eliminating redundant computations and reducing the overall verification time while maintaining class-discriminative precision.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent merges the computational operations by using the same forward propagation results for multiple class-specific backpropagation processes. Instead of performing separate forward and backward passes for each class, the system combines the forward computation results and applies them to generate multiple class-specific saliency maps through differentiated backpropagation, thereby reducing redundant computational overhead.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20220019870A1Verification of classification decisions in convolutional neural networks

Publication Date: 2022.01.20 SIEMENS AG

AI summary

In one aspect the invention relates to a computer-implemented method for providing a computer-implemented method for verifying a visual classification architecture of a convolutional neural network (CNN) and its decisions The method comprises to access (S1) a memory (MEM) with a convolutional neural network (CNN), being trained for a visual classification task into a set of target classes (tc); to use (S2) the convolutional neural network (CNN) for an input image (12) and after a forward pass of the convolutional neural network (CNN), in a backward pass: to apply (S3) a contrastive layer-wise relevance propagation algorithm (CLRP) or to apply (S4) a Bottom Up Attention pattern (BUAP), which is implicitly learned by the convolutional neural network (CNN) for providing (S5) a verification signal (vs).