Network Flow Partitioning via Time Buckets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network monitoring methodologies face challenges with high latency and resource contention due to the large volume of network flow information, which is exacerbated by sampling techniques that may not provide a complete picture of network traffic and lead to storage and processing capacity issues.
Innovation Solution
Organizing network flow information using a bucket identifier associated with each flow, derived from characteristics such as timestamps or network device identifiers, to facilitate efficient storage and analysis by partitioning data into sub-tables and aggregating records, allowing concurrent access and storage operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If network flow information is stored in a single table, then storage is simplified, but storage and processing capacity is exceeded and contention occurs
Solution Approach 1:
The patent divides the network flow information into multiple tables (raw data table, sub-tables, and aggregated data tables) based on time buckets. Each sub-table stores flow records for a specific time period, and aggregated data tables store summarized data. This segmentation allows the system to handle large volumes of data by distributing them across multiple tables rather than a single table, preventing storage capacity exhaustion and reducing contention.
2Productivity
If new network flow information is stored concurrently with analysis operations, then productivity increases, but storage resources are contended and access latency increases
Solution Approach 1:
By segmenting data into time-based sub-tables, the patent enables concurrent operations on different time periods. Analysis operations can query historical data in completed time buckets while new flow records are being stored in the current time bucket table. This temporal segmentation eliminates resource contention by isolating read and write operations to different data partitions, reducing access latency.
Solution Approach 2:
The system pre-organizes flow records into time buckets before storage, assigning each record to the appropriate sub-table based on its timestamp. This preliminary organization allows analysis operations to immediately access pre-partitioned data without waiting for storage operations to complete, enabling true concurrent operations and reducing latency.
3Quantity of substance
If sampling techniques are used to decrease data flow volume, then storage capacity issues are reduced, but network traffic picture becomes incomplete and measurement precision decreases
Solution Approach 1:
Instead of sampling individual packets or flows, the patent segments the entire data stream into time-based buckets and stores complete flow records in each bucket. This allows the system to maintain measurement precision by preserving all flow information while managing storage capacity through temporal partitioning and aggregation. The raw data table stores complete unsampled records, and aggregated tables provide summarized data for analysis.
Data Source
AI summary
To organize flows records into buckets and optimize their indexed access to/from storage medium, a bucket identifier is associated with each flow record. The bucket identifier is derived from some other characteristic of the network flow, such as a timestamp or source identification associated with the network flow. In this way, contention is decreased as analysis can occur on the aggregated flow records, while new flow records are stored.