Adaptive Hierarchical Heavy Hitter Detection in Network Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Detecting traffic anomalies in large packet-switched networks, such as VoIP networks, is challenging due to the high volume of data and complex hierarchical structures, making it difficult to maintain per-flow state and detect anomalies in near real-time.
Innovation Solution
A method and apparatus that use an adaptive data structure to identify hierarchical heavy hitters by estimating traffic aggregates and dynamically adjusting the granularity of monitoring, allowing for near real-time detection with lower update costs and deterministic accuracy, independent of data volume.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If per-flow state is maintained for anomaly detection, then detection accuracy is improved, but memory usage and processing overhead become prohibitive due to millions of concurrent flows
Solution Approach 1:
The patent segments the traffic monitoring problem into hierarchical levels (e.g., source IP, destination IP, protocol, port combinations). Instead of tracking all individual flows, it monitors aggregates at multiple hierarchical levels, allowing detection of anomalies at appropriate granularity without requiring per-flow state for all traffic
Solution Approach 2:
The patent introduces hierarchical aggregation levels as an additional dimension to the monitoring approach. By organizing traffic statistics in a hierarchical structure (from coarse-grained aggregates to fine-grained flows), it enables accurate anomaly detection at relevant levels while avoiding the memory burden of tracking every individual flow
2Reliability
If all traffic components are tracked individually, then detection completeness is improved, but processing time and computational resources become infeasible
Solution Approach 1:
The patent divides the traffic stream into hierarchical segments, processing and aggregating traffic statistics at multiple levels. This segmentation allows the system to identify anomalies at aggregate levels quickly, without requiring complete individual inspection of every packet or flow, thus maintaining detection completeness for significant anomalies while reducing processing time
Solution Approach 2:
The patent applies partial action by focusing computational resources on identifying and monitoring hierarchical aggregates that are likely to contain anomalies, rather than uniformly processing all traffic components. This allows the system to achieve sufficient detection completeness for security and performance monitoring without the excessive processing cost of complete individual flow analysis
3Measurement precision
If monitoring granularity is increased to detect all hierarchical anomalies, then detection accuracy is improved, but computational complexity increases significantly
Solution Approach 1:
The patent segments the monitoring task across hierarchical levels, where each level handles a specific granularity of traffic aggregation. This segmentation allows accurate detection of anomalies at different hierarchical levels (e.g., source IP level, destination IP level, flow level) without requiring the entire system to operate at maximum granularity, thus reducing overall computational complexity while maintaining detection accuracy where needed
Data Source
AI summary
An efficient streaming method and apparatus for detecting hierarchical heavy hitters from massive data streams is disclosed. In one embodiment, the method enables near real time detection of anomaly behavior in networks.