Multivariate Anomaly Detection via CUSUM Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection methods are inadequate for real-time, high-dimensional, and heterogeneous data systems, particularly in scenarios like cybersecurity and IoT networks, as they fail to accurately and quickly detect anomalies and provide effective localization, especially in cases of stealthy cyberattacks and changes in correlation structures.
Innovation Solution
A nonparametric, semi-supervised online multivariate anomaly detection method using k-NN distances to estimate the most compact set of nominal data points, allowing for real-time detection and localization of anomalies, with an extension to supervised settings using additional anomaly data for improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multivariate anomaly detection is used to capture anomaly evidence in interactions between system dimensions, then detection accuracy is improved, but computational complexity increases due to high-dimensionality
Solution Approach 1:
The patent segments the high-dimensional anomaly detection problem into multiple univariate CUSUM detectors, each monitoring a specific dimension independently. This segmentation reduces computational complexity from exponential in high dimensions to linear, while the fusion of evidence from multiple segments maintains detection accuracy for multivariate anomalies.
Solution Approach 2:
The patent transforms the high-dimensional multivariate detection problem into a low-dimensional space by using the fusion rule that combines p-values from univariate detectors. This dimensionality transformation allows accurate multivariate anomaly detection without the computational burden of directly processing high-dimensional data.
2Loss of time
If real-time processing of massive heterogeneous data is implemented, then timely anomaly detection is improved, but system resource consumption increases
Solution Approach 1:
The patent implements a lightweight anomaly detection system that processes data streams efficiently using simple CUSUM statistics and p-value fusion. The system serves itself by requiring minimal computational resources while maintaining real-time processing capability, making it suitable for deployment in resource-constrained environments.
3Loss of information
If anomaly localization is provided to explain where anomaly occurred, then interpretability is improved, but detection complexity increases
Solution Approach 1:
The patent performs anomaly localization as a preliminary action by identifying which dimensions contribute to the anomaly signal through the CUSUM statistics. By pre-computing dimension-wise statistics and using them for both detection and localization, the system provides interpretability without requiring separate complex localization algorithms.
Data Source
AI summary
A system and method for real-time detection of anomalies in high-dimensional systems. Various embodiments detect anomalies quickly and accurately to allow adequate time for appropriate countermeasures to be implemented. The sequential and multivariate anomaly detection method scales well to high-dimensional datasets using a nonparametric and semi-supervised approach that trains only on nominal data.