Anomaly Detection Mechanism Using Optimized ESD Test
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Customer Relationship Management (CRM) systems face challenges in efficiently detecting anomalies in high-volume data streams, which can impact system performance, and existing methods are computationally expensive and inefficient.
Innovation Solution
The implementation of an optimized Extreme Studentized Deviate (ESD) test that reduces memory requirements and computational costs by using a data structure with min-heap and max-heap to store largest and smallest values, allowing for real-time anomaly detection without storing all data points, enabling simultaneous processing of multiple data streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional anomaly detection methods are used on high-volume data streams, then anomaly detection capability is achieved, but computational cost and memory requirements become excessively high
Solution Approach 1:
The patent extracts only the essential information needed for anomaly detection by maintaining running statistics (mean, standard deviation, count) and using heaps to store only extreme values, rather than processing or storing all raw data points. This extraction approach reduces computational load while preserving anomaly detection capability.
Solution Approach 2:
The data stream is segmented into processing windows or batches, allowing incremental computation of statistics. The algorithm processes data in segments rather than requiring all data to be loaded into memory simultaneously, reducing memory requirements while maintaining detection accuracy.
2Measurement precision
If all data points are stored for anomaly detection, then complete analysis is possible, but memory usage increases significantly
Solution Approach 1:
The patent extracts only the necessary statistical parameters (mean, standard deviation, count) and extreme values needed for anomaly detection, discarding redundant data. This extraction maintains detection accuracy while dramatically reducing memory requirements from storing all data points to storing only aggregated statistics.
Solution Approach 2:
Instead of storing all data and then analyzing, the patent inverts the approach by computing and maintaining running statistics incrementally as data flows through, and using heaps to retain only the most relevant extreme values. This inversion enables continuous anomaly detection with minimal memory footprint.
3Measurement precision
If comprehensive data is processed for anomaly detection, then detection accuracy is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary computation of running statistics (mean, standard deviation, count) as data arrives, preparing the necessary parameters for anomaly detection in advance. This preliminary action enables rapid anomaly detection decisions without requiring repeated computations on raw data, reducing processing time while maintaining accuracy.
Data Source
AI summary
Techniques and structures to facilitate anomaly detection, including receiving performance metric data at a database system having a plurality of data point values, processing the plurality of data point values via an optimized Extreme Studentized Deviate (ESD) test to detect one or more anomalous data point values within the plurality of data point values, including storing a first set of values representing the largest values of the plurality of data point values and a second set of values representing the smallest values of the plurality of data point values, receiving a first of the plurality of data point values, determining whether the first data point value is to be included in the first set of values or the second set of values and determining that the first data point value is not an anomalous data point value upon a determination that the first data point value is not to be inserted into the first set of values or the second set of values and generating an incident alert upon a detection of one or more anomalous data point values within the plurality of data point values.