Adaptive Sketch-Set Aggregation for Network Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Monitoring data flow across high bandwidth networks is challenging due to increasing traffic volume, requiring large sets of detailed flow records for anomaly detection, which is inefficient and may lead to false positives and negatives.
Innovation Solution
An adaptive and iterative approach using data-created sketch-sets, where monitors collect and process data locally, and an aggregator combines and refines sketch-sets to detect anomalies with a feedback mechanism requesting finer data sets as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large sets of detailed flow records are collected for anomaly detection, then measurement precision is improved, but device complexity and processing costs increase
Solution Approach 1:
The patent segments the monitoring system into multiple distributed monitors that each collect local flow records independently, rather than using a single centralized system. Each monitor creates sketch-sets from its local data, which are then aggregated. This segmentation reduces the complexity burden on any single device while maintaining overall detection precision through combined analysis.
Solution Approach 2:
The patent introduces sketch-sets as an intermediary data structure between raw flow records and anomaly detection results. Instead of directly processing large detailed flow records, the system transforms them into compressed sketch representations that retain essential statistical properties. This intermediary layer reduces processing complexity while preserving the information needed for accurate anomaly detection.
2Measurement precision
If large sets of detailed flow records are collected for anomaly detection, then measurement precision is improved, but memory consumption increases
Solution Approach 1:
The patent extracts only the essential statistical features from detailed flow records to create sketch-sets, discarding redundant information. Each sketch-set contains aggregated statistics (such as packet counts, byte counts, and flow characteristics) that are sufficient for anomaly detection without storing complete detailed records. This extraction process significantly reduces memory consumption while maintaining detection precision.
Solution Approach 2:
The patent changes the parameter representation from detailed individual flow records to aggregated statistical parameters in sketch-sets. By transforming the data from fine-grained to coarse-grained parameters, the system reduces the quantity of data stored in memory while preserving the key characteristics needed for anomaly detection through mathematical aggregation and feedback mechanisms.
3Measurement precision
If large sets of detailed flow records are collected for anomaly detection, then measurement precision is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary processing of flow records at distributed monitors before aggregation, creating sketch-sets that pre-compute essential statistics. This preliminary action at the edge devices reduces the amount of work needed during centralized aggregation and analysis, thereby reducing overall processing time while maintaining detection precision through the pre-prepared statistical summaries.
Solution Approach 2:
The patent uses feedback mechanisms to request finer sketch-sets only when anomalies are detected or suspected, rather than continuously processing all data at maximum detail. This partial action approach processes data at lower detail levels during normal operation, reducing processing time, and only intensifies processing when necessary to maintain precision for actual anomalies.
4Quantity of substance
If traditional monitoring approaches are used, then comprehensive data collection is achieved, but false positives and negatives increase
Solution Approach 1:
The patent implements feedback mechanisms where the aggregator analyzes combined sketch-sets and requests refined or additional sketch-sets from specific monitors when anomalies are detected. This feedback loop allows the system to focus detailed data collection on suspicious activities rather than uniformly collecting all data, reducing false positives by verifying anomalies through targeted feedback-driven data requests while maintaining comprehensive coverage where needed.
Data Source
AI summary
A method for detecting an anomaly in a network can include combining a number of data-created sketch-sets and requesting a finer sketch-set for an identified sketch-set among the combined number of sketch-sets using an aggregator, and creating the finer sketch-set for the identified sketch-set to detect the anomaly in the network using a monitor.