Anomaly Detection in Data Streams via Median Deviation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection methods in complex systems, such as networks, are slow to detect shorter duration anomalies due to reliance on Fourier transforms, leading to delayed detection and temporary service disruptions during cyber attacks, and are prone to false positives, which can impact business operations and user access.
Innovation Solution
A method and apparatus that collect and process data streams into time intervals, calculating deviations from median values to quickly identify anomalies, reducing false positives and improving detection speed by using a collector, profiler, and detector module to analyze structured, unstructured, or hybrid data streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Fourier transforms are used to establish a baseline for anomaly detection, then measurement precision is improved, but detection speed deteriorates with delays of 10-15 minutes or longer
Solution Approach 1:
The patent segments the baseline calculation into multiple frequency components using Fourier transforms, allowing different parts of the frequency spectrum to be analyzed separately. This enables selective monitoring of high-frequency components that indicate anomalies while maintaining overall detection accuracy, thereby reducing the time required for complete baseline analysis.
Solution Approach 2:
The patent applies partial action by focusing detection efforts primarily on high-frequency components where anomalies are most likely to appear, rather than analyzing the entire frequency spectrum equally. This selective approach maintains measurement precision for anomaly detection while significantly reducing the computational time and detection delay.
2Reliability
If traffic is diverted to a sink hole for dDoS scrubbing, then protection from cyber attacks is improved, but service availability deteriorates due to black-hole effects
Solution Approach 1:
The patent implements dynamic routing that can adapt in real-time based on anomaly detection results. Rather than permanently diverting traffic to sink holes, the system dynamically adjusts routing decisions, allowing traffic to be restored to normal paths when attacks cease, thereby maintaining service availability while providing protection during attack periods.
Solution Approach 2:
The patent introduces an intermediary anomaly detection and decision-making system between the network traffic and the sink hole routing. This intermediary evaluates incoming traffic and makes intelligent routing decisions, diverting only malicious traffic to sink holes while allowing legitimate traffic to continue normally, thus protecting against attacks without causing black-hole effects on legitimate services.
3Measurement precision
If the detection system monitors all frequency components, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent extracts and focuses specifically on high-frequency components of the network traffic spectrum, separating them from the full frequency spectrum. By extracting only the relevant high-frequency portions where anomalies manifest, the system maintains measurement precision for anomaly detection while significantly reducing the complexity of processing the entire data set.
Data Source
AI summary
There is provided a method for detecting an anomaly in plurality of data streams originating from a system or network of systems. Data streams are collected from the system or systems and divided into a plurality of time intervals. For each of the plurality of time intervals, a value for a parameter associated with the data stream is determined. A deviation in the determined values is calculated for the parameters associated with the data stream from expected values for the parameters and, if the calculated deviation is above a threshold, an anomaly is detected in the collected data stream.