Data Anomaly Detection Using Interpretable Invariant Rules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods, particularly those using deep neural networks, lack interpretability, making it difficult for human experts to understand why specific data instances are identified as anomalies, which is crucial in applications like system health monitoring.
Innovation Solution
A method for detecting data anomalies by generating invariant data rules based on historical data, using antecedent and consequent predicate sets, where the intersection is empty, ensuring accurate and interpretable anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are used for anomaly detection, then detection accuracy is improved, but interpretability deteriorates
Solution Approach 1:
The patent segments the anomaly detection process into two distinct parts: (1) using deep neural networks for accurate anomaly detection, and (2) separately generating interpretable explanations through rule extraction. This segmentation allows each component to optimize for its specific function without compromising the other.
Solution Approach 2:
The patent introduces an intermediary component that translates the black-box neural network decisions into human-understandable rules. This intermediary layer extracts meaningful patterns from network outputs and formulates them as interpretable detection rules, bridging the gap between accurate but opaque neural networks and human need for explanation.
2Reliability
If complex deep learning models are deployed, then detection capability is improved, but system complexity increases
Solution Approach 1:
The patent extracts the essential detection logic from complex deep learning models and formulates it as simpler, more manageable rules. By taking out the core detection capability and representing it in a rule-based format, the system maintains high detection reliability while reducing the operational complexity of deploying and maintaining the model.
Solution Approach 2:
The patent transforms the parameter representation from high-dimensional neural network weights and activations to human-readable rule parameters with clear semantic meanings. This parameter transformation maintains the detection capability while making the system more interpretable and easier to manage.
3Measurement precision
If more historical data is processed to establish rules, then rule accuracy is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary processing of historical data to pre-extract meaningful patterns and statistics before actual anomaly detection. By preparing the data in advance and establishing baseline rules from historical patterns, the system can quickly detect anomalies without reprocessing all historical data each time, thus improving rule accuracy while controlling processing time.
Data Source
AI summary
Various embodiments of the teachings herein include a method for detecting data anomalies. A method may include: receiving test data; and matching the test data with a data rule determined on the basis of historical data having a shared data type with the test data. The data rule contains an antecedent and a consequent predicate set. An intersection of the antecedent predicate set and the consequent predicate set is an empty set. The antecedent predicate set contains at least one antecedent predicate. The consequent predicate set contains at least one consequent predicate. When the data to be tested satisfies all the antecedent predicates in the antecedent predicate set and fails to satisfy at least one consequent predicate in the consequent predicate set, the test data is flagged as anomalous due to failure to satisfy the data rule.