Industrial ML Output Explanation Using Input Perturbation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models in industrial applications, such as Deep Learning networks, are often 'black box' models, making it difficult to trust their output and diagnose the cause of unreliable predictions, limiting their usefulness due to a lack of insight into their reasoning process.
Innovation Solution
An explainer system that applies predetermined perturbations to original sample data to produce perturbed data, which is then used to identify deviations in model output, allowing the extraction of important features to explain how the model arrived at its predictions, thereby providing insights into the source and nature of predicted issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If black box machine learning models (Deep Learning networks, Support Vector Machines, ensemble methods) are used to achieve good performance with few false positives and false negatives, then prediction accuracy is improved, but the output becomes untrustworthy and difficult to verify because the reasoning process is not transparent
Solution Approach 1:
The patent introduces an explainer system as an intermediary between the black box ML model and human operators. This explainer system receives the model output and generates human-understandable explanations by analyzing the relationship between input data and model predictions, thereby mediating the trust gap without modifying the original high-accuracy model
Solution Approach 2:
The explainer system creates a simplified copy or representation of the black box model's reasoning process. Instead of trying to open the black box, it generates a parallel explanation model that replicates the decision-making logic in human-comprehensible terms, allowing verification without sacrificing the original model's predictive performance
2Productivity
If black box machine learning models are deployed in industrial applications, then productivity is improved through automated quality checks and anomaly detection, but the lack of insight into the model's reasoning limits its usefulness and requires further investigation to diagnose unreliable output
Solution Approach 1:
The explainer system acts as an intermediary layer that translates the black box model's automated decisions into human-understandable explanations. This allows the system to maintain high automated productivity while simultaneously providing diagnostic insights that make the system easy to operate and troubleshoot when needed
3Measurement precision
If engineered features are used during the training process to achieve good model performance, then prediction accuracy is improved, but the features become hard to comprehend for human operators who need to understand and verify the model output
Solution Approach 1:
Instead of trying to make engineered features more comprehensible by modifying them, the patent inverts the approach: it takes the already-trained model with optimized engineered features and builds an explainer that translates the model's internal reasoning back into human-understandable terms. This preserves the high performance achieved through feature engineering while restoring comprehensibility through explanation generation
Data Source
AI summary
An explainer system includes a system-monitor machine learning model trained to predict states of a monitored system, a perturbator applying predetermined perturbations to original sample data collected from the monitored system to produce perturbed sample data. The system is configured to input the perturbed sample data to the prediction system. The explainer comprises a tester that receives model output from the prediction system, the model output comprising original model output produced by the system-monitor machine learning model based on the original sample data and deviated model output produced by the system-monitor machine learning model based on the perturbed sample data, the deviated model output comprising deviations from the original model output, the deviations resulting from the applied perturbations. An extractor receives data defining the perturbations and the resulting deviations and extracts therefrom important features for explaining the model output.