KPI Clustering for Scalable Neural Network Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for characterizing computerized systems struggle with inefficiency and scalability in anomaly detection, particularly in large datasets, and fail to provide deep root cause analysis due to inefficiency in explainability and causality methods.
Innovation Solution
A method that channels key performance indicators (KPIs) through buffer channels, clusters them into groups, identifies representative KPIs, and feeds them into a trained neural network for characterization, allowing for efficient anomaly detection and root cause analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all available KPIs are ingested systematically into the neural network, then complete information is preserved for characterization, but the processing efficiency and scalability deteriorate significantly on large datasets
Solution Approach 1:
The patent segments the large set of KPIs into multiple clusters based on similarity metrics. Each cluster contains KPIs with related characteristics or behaviors. This segmentation allows the system to process clusters independently and selectively, rather than processing all KPIs uniformly, thereby improving processing efficiency while preserving important information through representative selection from each cluster.
Solution Approach 2:
The patent extracts representative KPIs from each cluster based on their contribution to anomaly detection and system characterization. By selecting only the most informative KPIs from each cluster (rather than including all KPIs), the system reduces the input dimensionality to the neural network, improving processing efficiency and scalability while maintaining the essential information needed for effective anomaly detection.
2Ease of operation
If explainability methods are applied to analyze system behavior, then interpretability is improved, but the ability to perform deep root cause analysis deteriorates due to inefficiency
Solution Approach 1:
The patent performs preliminary clustering and representative KPI selection before feeding data into the neural network. This preliminary action organizes the KPIs in a way that preserves interpretability (through structured clustering) while enabling more efficient and deeper root cause analysis (by focusing on representative KPIs that capture essential system behaviors). The clustering structure itself provides interpretability, and the representative selection enables deeper analysis without the inefficiency of processing all raw KPIs.
3Measurement precision
If baseline causality methods are run on large datasets to perform root cause analysis, then causal inference is improved, but scalability and efficiency deteriorate
Solution Approach 1:
The patent segments the large dataset of KPIs into manageable clusters, allowing causality methods to be applied to smaller, more focused subsets of related KPIs. This segmentation maintains causal inference accuracy by preserving the relationships within each cluster while improving scalability by reducing the overall computational burden. The clustered structure enables parallel processing and more efficient causal analysis across the entire system.
Solution Approach 2:
The patent extracts representative KPIs from each cluster that are most relevant to causal inference. By selecting only the key representative KPIs from each cluster (rather than analyzing all KPIs), the system maintains the precision of causal inference on critical system behaviors while significantly improving scalability and efficiency when analyzing large datasets. This extraction focuses computational resources on the most informative signals for root cause analysis.
Data Source
AI summary
The invention is directed to characterizing a computerized system based on key performance indicators (KPIs). Channel the KPIs through n buffer channels, where n≥2. The KPIs are channeled by: accessing KPIs; clustering the KPIs accessed to obtain k clusters; and, for each cluster of the k clusters obtained, identifying n representative KPIs in each cluster as objects of n respective types and buffering KPI values of the n representative KPIs identified in respective ones of the n buffer channels. The KPI values buffered in the n buffer channels are then fed into respective input channels of a trained neural network. The neural network independently processes the KPIs in the input channels and produces output values in output of the inner layers. Eventually, the computerized system is characterized based on the output values produced, e.g., to detect an anomaly in the system.