Network Anomaly Detection via Intermediate Trend Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network anomaly detection systems require computationally expensive algorithms that consume large amounts of memory, making them inefficient for networks where denial-of-service (DoS) attacks are rare, as they continuously process network flow data without focusing on potential anomalies efficiently.
Innovation Solution
An intermediate processing stage is introduced that performs low-complexity computations on network flow data to detect potential anomalies by comparing short-term and long-term trends, initiating more robust algorithms only when anomalies are likely, thereby conserving computational and memory resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If robust anomaly detection algorithms are continuously executed, then detection reliability is improved, but computational resource consumption increases
Solution Approach 1:
The system performs preliminary filtering by computing basic statistics (packet count, byte count) and detecting potential anomalies using simple threshold comparisons before invoking complex anomaly detection algorithms. This preliminary action identifies candidate anomalies that warrant further investigation, avoiding unnecessary execution of computationally expensive algorithms during normal network conditions.
Solution Approach 2:
The anomaly detection process is segmented into multiple stages: (1) basic statistic computation stage, (2) potential anomaly detection stage, and (3) robust algorithm execution stage. Each stage processes data with increasing complexity only when necessary, dividing the overall detection task into manageable segments that optimize resource usage while maintaining detection reliability.
2Measurement precision
If robust anomaly detection algorithms are continuously executed, then detection precision is improved, but memory consumption increases
Solution Approach 1:
The system computes basic statistics and identifies potential anomalies using minimal memory before invoking memory-intensive robust algorithms. This preliminary action filters out normal traffic patterns that would otherwise consume unnecessary memory resources during continuous execution of complex detection algorithms.
Solution Approach 2:
The system applies different processing quality levels to different network traffic conditions: simple statistic computation for normal traffic and robust algorithm execution only for suspicious patterns. This local quality approach ensures high detection precision is applied only where needed, optimizing the balance between precision and memory consumption.
3Productivity
If intermediate processing stage is added, then resource efficiency is improved, but device complexity increases
Solution Approach 1:
The processing system is segmented into distinct functional modules: a first module for computing basic statistics, a second module for detecting potential anomalies, and a third module for executing robust algorithms. This segmentation organizes complexity into manageable, independently executable components that improve resource efficiency while maintaining clear system architecture.
Data Source
AI summary
In an embodiment, a computer implemented method receives flow data for a network flows. The method extracts a tuple from the flow data and calculates long-term and short-term trends based at least in part on the tuple. The long-term and short-term trends are compared to determine whether a potential network anomaly exists. If a potential network anomaly does exist, the method initiates a heavy hitter detection algorithm. The method forms a low-complexity intermediate stage of processing that enables a high-complexity heavy hitter detection algorithm to execute when heavy hitters are likely to be detected.