Deep Learning Anomaly Detection with Explanation Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network assurance systems face challenges in accurately detecting and explaining behavioral anomalies in complex computer networks, often resulting in irrelevant alerts due to the assumption that rare observations are relevant, and lack of contextual features leading to false positives.
Innovation Solution
A deep learning architecture for collaborative anomaly detection and explanation is introduced, utilizing separate contextual and explanatory features to train anomaly detectors and explanation models end-to-end, ensuring that only explainable anomalies trigger alerts, thereby reducing false positives and providing actionable insights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional anomaly detection methods are used that assume rare observations are relevant, then anomaly detection capability is provided, but false positives increase and precision decreases
Solution Approach 1:
The patent introduces an explanation model as an intermediary between the anomaly detector and the alerting system. This explanation model validates detected anomalies by providing contextual explanations, filtering out false positives before they become alerts. The explanation model acts as a mediator that verifies the legitimacy of anomaly detections using additional contextual features.
Solution Approach 2:
The system implements feedback loops where explanation models validate anomaly detections and provide feedback to refine the detection process. The collaborative training framework uses feedback from both anomaly detection outcomes and explanation quality to continuously improve the system's precision in identifying true anomalies versus false positives.
2Productivity
If only anomaly detection is performed without explanation models, then detection speed is maintained, but recall decreases due to lack of contextual understanding
Solution Approach 1:
The system performs preliminary actions by pre-training explanation models with contextual features before actual anomaly detection. This preliminary preparation enables the explanation models to quickly validate anomalies during runtime without significantly impacting detection speed, while simultaneously improving recall through better contextual understanding.
Solution Approach 2:
The patent merges anomaly detection and explanation generation into a unified collaborative framework. Both functions operate together in an end-to-end training process, allowing the system to maintain detection speed while improving recall through the combined power of anomaly detectors and explanation models working synergistically.
3Measurement precision
If separate contextual and explanatory features are used to train models end-to-end, then precision and recall are enhanced, but device complexity increases
Solution Approach 1:
The patent segments the feature space into distinct contextual features and explanatory features, each trained by specialized models. This segmentation allows for more focused and efficient training processes compared to trying to learn all features in a single monolithic model, thereby managing complexity while improving performance.
Solution Approach 2:
The collaborative training framework creates a universal system where multiple models (anomaly detectors and explanation models) work together to achieve multiple objectives simultaneously. This multi-functional approach enhances precision and recall while distributing complexity across specialized components rather than concentrating it in a single complex model.
4Device complexity
If traditional anomaly detection without explanation is used, then system simplicity is maintained, but loss of information increases due to lack of root cause identification
Solution Approach 1:
Explanation models serve as intermediaries that bridge the gap between anomaly detection and root cause identification. These models process contextual features to generate human-understandable explanations, preserving critical information about anomaly causes without requiring complete system redesign.
Solution Approach 2:
The system creates explanatory representations (copies) of the underlying causes from complex network data. Instead of directly analyzing raw network data for root causes, the explanation models generate simplified copies or representations of causal relationships that are easier to interpret while retaining essential information.
Data Source
AI summary
In one embodiment, a network assurance service that monitors a network detects a behavioral anomaly in the network using an anomaly detector that compares an anomaly detection threshold to a target value calculated based on a first set of one or more measurements from the network. The service uses an explanation model to predict when the anomaly detector will detect anomalies. The explanation model takes as input a second set of one or more measurements from the network that differs from the first set. The service determines that the detected anomaly is explainable, based on the explanation model correctly predicting the detection of the anomaly by the anomaly detector. The service provides an anomaly detection alert for the detected anomaly to a user interface, based on the detected anomaly being explainable. The anomaly detection alert indicates at least one measurement from the second set as an explanation for the anomaly.