Interpretable Anomaly Detection via Invariant Data Rules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods, particularly those using deep neural networks, fail to provide interpretable reasons for identifying data anomalies, which is crucial for practical applications like system health monitoring in factories.
Innovation Solution
A method and apparatus for detecting data anomalies by generating invariant data rules from historical data, dividing the closed frequent predicate set into antecedent and consequent predicate sets, and ensuring statistical significance through support operations, enabling accurate and interpretable anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are used for anomaly detection, then detection accuracy is improved, but interpretability of anomaly reasons deteriorates
Solution Approach 1:
The patent segments the anomaly detection process into two distinct components: (1) a deep neural network for accurate anomaly detection, and (2) a separate rule generation module that creates interpretable rules from historical data. This segmentation allows each component to specialize - the neural network focuses on accuracy while the rule-based system provides interpretability, resolving the contradiction between detection accuracy and reason interpretability
Solution Approach 2:
The patent introduces an intermediary rule generation mechanism that bridges the black-box neural network and the need for interpretable explanations. This intermediary component generates rules from historical data that can explain why certain anomalies are detected, serving as a mediator between the accurate but opaque neural network and the requirement for transparent reasoning
2Loss of information
If invariant rules are generated from historical data, then interpretability is improved, but detection accuracy may deteriorate
Solution Approach 1:
The patent merges two complementary approaches - deep neural networks and invariant rule generation - into a unified anomaly detection system. The neural network component captures complex patterns for high accuracy, while the rule-based component provides interpretable constraints. By combining these approaches, the system achieves both detection accuracy and interpretability, as each component compensates for the other's weaknesses
3Reliability
If complex data processing is performed to generate predicates, then anomaly detection capability is improved, but system complexity deteriorates
Solution Approach 1:
The patent performs preliminary action by pre-generating invariant rules and predicates from historical data before actual anomaly detection occurs. This pre-processing creates a ready-to-use rule base that simplifies the real-time detection process, as the complex predicate generation work is completed in advance rather than during operational anomaly detection
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present invention disclose a method, an apparatus, an electronic device, and a readable storage medium for detecting data anomalies. The method comprises receiving data to be tested; matching the data to be tested with a data rule determined on the basis of historical data, the historical data having the same data type as the data to be tested, the data rule containing an antecedent predicate set and a consequent predicate set, wherein the intersection of the antecedent predicate set and the consequent predicate set is an empty set, the antecedent predicate set contains at least one antecedent predicate, and the consequent predicate set contains at least one consequent predicate; and, when the data to be tested satisfies all the antecedent predicates in the antecedent predicate set and fails to satisfy at least one consequent predicate in the consequent predicate set, determining that the data to be tested is anomalous due to failure to satisfy the data rule. With embodiments of the present invention, data anomalies are accurately detected, and reasons for the occurrence of anomalies are also interpretable.