Black-Box Model Explanation via Embedding Space Perturbation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for explaining data analysis models, particularly black-box models, face challenges in practicability due to their complex structures, making it difficult to understand the importance of features and interactions, and suffer from poor stability in explanation results.
Innovation Solution
The method transforms a target input sample into an embedding space, generates a perturbation dataset, determines weights of neighborhood vectors based on distance, and trains an explainable model using these vectors and output vectors to acquire feature weights, improving practicability and stability by using an autoencoder and transforming data between feature and embedding spaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a black-box data analysis model is used to improve prediction accuracy, then the model's analytical capability is improved, but the model structure becomes complex and difficult to explain
Solution Approach 1:
The patent introduces an embedding space as an intermediary representation between the original feature space and the model's internal representations. This embedding space serves as a mediator that allows for the generation of neighborhood perturbation data while maintaining a clear relationship to the original features, thereby enabling explanation of complex black-box models without sacrificing prediction accuracy.
Solution Approach 2:
The patent transforms the problem from the original feature space to an embedding space, effectively changing the dimensionality and representation of the data. This dimensional transformation allows for more effective generation of neighborhood perturbations and facilitates the training of explainable models that can capture feature importances in a more tractable space.
2Ease of operation
If traditional explanation methods are used to simplify model interpretation, then the explanation process becomes easier, but the stability of explanation results deteriorates
Solution Approach 1:
The patent performs preliminary transformation of the input sample into an embedding space before generating perturbations. This preliminary action in the embedding space creates a more stable foundation for subsequent perturbation generation, leading to more stable explanation results while maintaining ease of operation through the structured approach.
Solution Approach 2:
The patent employs a feedback mechanism where the explainable model is trained using perturbation data and output vectors, and the generated explanations can be used to refine and validate the model's feature importance assessments. This iterative feedback process improves the stability of explanation results while keeping the explanation process systematic and manageable.
3Ease of operation
If feature weights are determined directly from complex model outputs, then the explanation process is simplified, but the accuracy of feature importance measurement deteriorates
Solution Approach 1:
The patent performs preliminary transformation of features into an embedding space and generates neighborhood perturbations before determining feature weights. This preliminary processing in the embedding space ensures that the feature importance measurements are based on more accurate and meaningful representations, improving measurement precision while maintaining a systematic approach.
Solution Approach 2:
The patent replaces direct extraction of feature weights from complex model outputs with a training-based approach using an explainable model. Instead of mechanically extracting weights from black-box outputs, the system uses a learning process that trains an explainable model on perturbation data, providing more accurate feature importance measurements through adaptive learning rather than direct mechanical extraction.
Data Source
AI summary
Disclosed is a method for explaining a model. The method comprises: transforming a target input sample from an original feature space to an embedding space; generating a perturbation data set in the embedding space; acquiring a weight of a neighborhood vector in the perturbation data set; after that, transforming the perturbation data set back to the original feature space; acquiring an explainable model of a target data analysis model by training based on the acquired data; and acquiring an explanation result based on the explainable model.