Distributed Flowtree Summarization for Scalable Network Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network flow capture systems face challenges in efficiently analyzing massive data from large networks spanning multiple time periods, as they require significant storage, processing capabilities, and manual, slow query processes, while current summaries lack comprehensive insights and are not scalable for real-time and historical analysis.
Innovation Solution
A distributed system using a self-adjusting Flowtree data structure that maintains hierarchical summaries of network flows, allowing for prompt and accurate query execution across sites and time, with operators like merge, compress, diff, and top-K, reducing storage requirements and enabling interactive network analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If flow captures are collected at multiple border and backbone routers, then network monitoring coverage is improved, but data transfer cost and complexity increase
Solution Approach 1:
The patent divides the network into multiple autonomous domains, each maintaining its own flow capture data locally. This segmentation eliminates the need to transfer all raw flow data across the entire network, as each domain processes and analyzes its own data independently, thereby reducing data transfer costs while maintaining comprehensive monitoring coverage.
Solution Approach 2:
The patent introduces a hierarchical dimension to flow capture analysis, moving from flat centralized processing to multi-level distributed processing. Flow data is first processed at the domain level, then aggregated and analyzed at higher hierarchical levels only when necessary, reducing the volume of data that needs to be transferred across the network while preserving monitoring effectiveness.
2Quantity of substance
If summaries of only the most popular 5-tuple flows are kept, then storage requirements are reduced, but information on less popular ports and prefixes is lost
Solution Approach 1:
The patent applies different levels of summarization quality to different types of flow data based on their importance. Popular 5-tuple flows maintain detailed information, while less popular flows are summarized at coarser granularities. This local quality differentiation reduces overall storage requirements while preserving critical information about less popular ports and prefixes that might be relevant for security or anomaly detection.
Solution Approach 2:
The patent implements dynamic summarization where the level of detail stored for flows adapts based on their observed characteristics and importance. Flows that show signs of being unusual, security-relevant, or potentially important are maintained with higher detail, while clearly normal and popular flows use more aggressive summarization, optimizing the balance between storage and information retention.
3Duration of action of stationary object
If raw flow captures are stored for long time windows, then historical analysis capability is improved, but storage costs increase prohibitively
Solution Approach 1:
The patent implements periodic aggregation and summarization of flow data at multiple time scales. Raw flow captures are maintained for short periods with high detail, then periodically aggregated into coarser summaries for longer time windows. This periodic transformation allows historical analysis capability to extend over long time windows while storage costs remain manageable due to the progressive coarsening of data representation.
4Productivity
If centralized processing location is used, then data analysis capability is improved, but transfer expense and regulatory compliance issues increase
Solution Approach 1:
The patent introduces domain-level aggregators and summary generators as intermediary components between distributed flow capture points and centralized analysis systems. These intermediaries perform preliminary processing and aggregation locally, transforming raw high-volume flow data into condensed summaries that can be efficiently transferred to centralized locations for advanced analysis, thereby reducing transfer expenses and bandwidth requirements.
Data Source
Figure 1
Figure 2(a)~2(b)
Figure 3(a)~3(b)
AI summary
Method and system for capturing summaries of data flows in an electronic, packet-switched communication network, the method comprising the steps: - receiving, from a node of the communication network, packet flow data; - obtaining a subset of the packet flow parameters (IP src, IP dst, port src, port dst, protocol), based on the packet flow data; - obtaining a packet flow statistic, based on the packet flow data; - updating a collection of stored packet flow summaries, based on a subset of the packet flow parameters and the packet flow statistic. According to the invention, the stored packet flow summaries are arranged as nodes in a tree data structure.