Sliding-Window Data Stream Detection With Statistical Alert Summaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing monitoring systems generate excessive alerts for insignificant anomalies, leading to alert fatigue, and fail to effectively detect ephemeral and significant changes in data streams from complex systems, which can compromise system health and data quality.
Innovation Solution
A system and method using sliding windows and Earth Mover's distance algorithm to compare data distributions, compute z-scores, and generate alerts when thresholds are exceeded, accompanied by a Generative AI for textual summaries to filter and prioritize alerts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional anomaly detection methods are used to monitor data streams, then anomalies can be detected, but excessive alerts are generated for insignificant anomalies leading to alert fatigue
Solution Approach 1:
The system performs preliminary actions by establishing baseline data distributions and computing statistical metrics (z-scores, p-values) in advance to determine significance thresholds. This allows the system to pre-configure what constitutes a significant anomaly before actual detection occurs, enabling automated filtering of insignificant alerts and reducing alert fatigue while maintaining reliable anomaly detection.
2Loss of information
If monitoring systems generate alerts for all anomalies, then all potential issues are identified, but operators cannot distinguish between severe alerts and minor glitches
Solution Approach 1:
The system applies local quality by differentiating alert severity based on statistical significance metrics computed for each specific anomaly instance. Instead of treating all anomalies uniformly, the system evaluates each anomaly's z-score and p-value relative to its local data distribution, assigning appropriate priority levels. This enables operators to easily distinguish between severe alerts (high z-scores, low p-values) and minor glitches, improving ease of operation while maintaining information completeness.
Solution Approach 2:
The system changes parameters by transforming raw anomaly data into standardized statistical metrics (z-scores, p-values) that enable quantitative comparison and prioritization. By converting diverse anomaly types into a common statistical framework, the system allows operators to uniformly assess and prioritize alerts based on objective criteria rather than subjective judgment, significantly improving ease of operation.
3Reliability
If the system detects all changes in data streams, then no significant anomalies are missed, but ephemeral anomalies are raised as alerts consuming operator attention
Solution Approach 1:
The system performs preliminary statistical analysis to establish baseline distributions and significance thresholds before actual anomaly detection. By pre-computing z-scores and p-value thresholds, the system can rapidly evaluate incoming anomalies against established criteria without time-consuming analysis, enabling immediate filtering of ephemeral anomalies while maintaining complete detection of significant changes, thus reducing operator response time.
Solution Approach 2:
The system implements feedback by continuously monitoring data distributions and updating baseline statistics as new data arrives. This adaptive feedback mechanism allows the system to learn normal behavior patterns over time and adjust its anomaly detection thresholds accordingly, improving the ability to distinguish between ephemeral fluctuations and genuine anomalies while maintaining detection completeness and reducing false alerts.
Data Source
AI summary
Aspects of the subject disclosure may include, for example, a device having a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations of: receiving a series of values from a data stream generated by one or more equipment in a communications network; creating a first window of values received before a first point in time and a second window of values received on or after the first point in time; comparing a distribution of values in the first window and values in the second window to compute a distance at the first point in time; repeating the creating and comparing at subsequent points in time after the first point in time, thereby generating a series of distances; computing a z-score for a first distance in the series of distances; generating an alert responsive to the z-score exceeding a threshold; and storing the alert in a log of alerts. Other embodiments are disclosed.