Anomaly Detection in Performance Management Using PCA Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing analytics systems face challenges in managing large datasets with millions of key performance indicators (KPIs), leading to unmanageable noise and false positive signals, and fail to effectively distinguish anomalies from normal behavior due to lack of temporal information.
Innovation Solution
Performing principal component analysis (PCA) to reduce the dimensionality of KPIs, clustering them in a transformed space, and classifying local and structural anomalies based on historical information to determine global or local anomalies, allowing for targeted management actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If simple clustering of raw KPIs is performed in a big data environment, then the system can process large numbers of KPIs, but the underlying noise generates strong false positive signals
Solution Approach 1:
The patent segments the massive set of KPIs into smaller, manageable clusters based on their correlations and relationships. By dividing the overall KPI space into distinct groups, the system can analyze each cluster separately, reducing the impact of noise from unrelated KPIs and improving anomaly detection precision while maintaining the ability to handle large quantities of KPIs
Solution Approach 2:
The patent transforms the analysis from the original high-dimensional KPI space to a new dimension defined by cluster centroids and distances. By measuring anomalies based on distance from cluster centroids rather than raw KPI values, the system creates a new dimensional framework that separates signal from noise, enabling accurate anomaly detection even with millions of KPIs
2Measurement precision
If anomaly detection is performed based on pure distance analysis without temporal information, then the system can identify deviations from clusters, but normal operation is treated as an anomaly
Solution Approach 1:
The patent performs preliminary actions by establishing temporal baselines and expected variation patterns for each cluster before detecting anomalies. The system pre-configures knowledge about normal operational variations and temporal patterns, so when distance-based deviations occur, they can be evaluated against these pre-established expectations to determine whether they represent true anomalies or normal fluctuations
Solution Approach 2:
The patent incorporates feedback mechanisms that use historical data and temporal patterns to continuously refine anomaly detection thresholds. By feeding back information about normal operational variations and temporal trends, the system adjusts its sensitivity to distance-based deviations, ensuring that normal operational changes are not misclassified as anomalies while maintaining detection of true anomalies
Data Source
AI summary
Methods and systems for detecting anomalous behavior include performing a principal component analysis on a plurality of key performance indicators (KPIs) to determine a set of principal axes. The KPIs are clustered in a space defined by the set of principal axes. Local and structural anomalies are determined in the clustered KPIs. The structural and local anomalies are classified based on historical information. A transformation is performed from a space based on the principal axes to an original space. It is determined whether each of the local and structural anomalies is a global or a local anomaly. A management action is performed based on the classified structural and local anomalies.