Adversarial Attack Explainability for Black-Box Machine Learning Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models, particularly deep neural networks, are opaque and lack explainability, making it difficult to understand their decision-making processes, which can lead to unjustifiable and illegitimate decisions.
Innovation Solution
The method involves executing an adversarial attack on a machine learning model to generate modified input data files, creating mappings of the changes, segmenting the data based on characteristics, determining the density of changes, and displaying the segments with significant changes via a graphical user interface to provide explainability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks with hundreds of layers and millions of parameters are used, then performance in classification, regression, and prediction tasks is improved, but explainability and interpretability of the decision-making process deteriorate
Solution Approach 1:
The patent introduces an intermediary system that acts as a bridge between the black-box deep neural network and human users. This intermediary takes the complex internal representations and decisions of the DNN and transforms them into human-interpretable explanations, allowing users to understand the decision-making process without compromising the model's high performance accuracy
Solution Approach 2:
The patent replaces the traditional mechanical approach of directly observing model internals with a symbolic reasoning system that generates natural language explanations. This substitution allows the system to maintain the computational power of deep neural networks while providing human-understandable interpretations through language rather than raw numerical data
2Productivity
If black-box models are used, then productivity and automation are improved, but the ability to detect errors and understand their sources deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where the explanation system continuously provides information about model decisions and their rationale. This feedback loop enables users to detect errors by understanding the reasoning behind predictions, identify when the model is making unjustifiable decisions, and trace the sources of errors back to specific input features or internal representations
Solution Approach 2:
The patent performs preliminary analysis by generating explanations before final decisions are acted upon. This allows users to review the reasoning and potential errors before the model's decisions are implemented, enabling preventive error detection and correction while maintaining automated productivity
3Measurement precision
If deep neural networks are used, then performance in complex tasks is improved, but the legitimacy and justifiability of decisions deteriorate
Solution Approach 1:
The patent introduces an intermediary explanation layer that translates the complex internal reasoning of deep neural networks into human-understandable justifications. This intermediary system generates narratives that explain why specific decisions were made, what factors were most influential, and how the model arrived at its conclusions, thereby restoring decision legitimacy and justifiability
Solution Approach 2:
The patent transforms the parameter space from raw numerical representations to human-interpretable concepts. By changing the representation parameters from abstract neural activations to meaningful feature attributions and logical reasoning steps, the system maintains high task performance while making decisions justifiable and explainable to human users
Data Source
AI summary
A method and system for explaining a decision process of a machine learning model that includes inputting into a machine learning model a first input data file; receiving a first output data file from the machine learning model based on the first input data file; executing an adversarial attack on the machine learning model, creating a mapping of the one or more units of data of the first input data file with changes by the adversarial attack exceeding a first threshold to one or more segments of the first input data file; determining a density of the changes to the one or more units of data in each of the one or more segments; and displaying the one or more segments of the first input data file having a density of changes to the one or more units of data exceeding a second threshold via a graphical user interface.