Network Flow Aggregation via Time-Partitioned Database Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network monitoring methodologies face challenges with large volumes of network flow information, leading to storage and processing capacity issues, and sampling techniques do not provide a complete picture of network traffic, causing data contention and ignoring important information.
Innovation Solution
The method involves aggregating network flow information within a relational database by storing raw flow records in separate tables based on time intervals, allowing concurrent analysis and storage, thereby minimizing contention and optimizing storage capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If network flow data is stored in a single database table, then data access is simple, but data contention increases when analysis and insertion occur simultaneously
Solution Approach 1:
The patent segments the database into multiple tables organized by time intervals (e.g., hourly, daily, weekly tables). This segmentation allows concurrent access patterns where analysis queries can access historical tables while new data is inserted into current tables, reducing data contention while maintaining operational simplicity through a systematic table structure.
2Quantity of substance
If sampling techniques are used to reduce data volume, then storage capacity requirements decrease, but complete network traffic picture is lost
Solution Approach 1:
The patent employs data aggregation techniques that create summarized representations of network flow data at various time intervals. These aggregated tables store condensed information (e.g., total bytes, packet counts, average values) that capture essential network traffic patterns without requiring storage of every individual flow record, effectively reducing data volume while preserving meaningful information.
3Measurement precision
If all network flow records are stored in detail, then complete analysis is possible, but storage and processing capacity requirements increase
Solution Approach 1:
The patent implements a dynamic data retention strategy where the level of data detail is adjusted based on time. Recent data is maintained in detailed format in current tables for precise analysis, while historical data is progressively aggregated into summarized tables. This dynamic approach allows complete analysis capability for recent data while reducing storage requirements for historical data through time-based aggregation.
Data Source
AI summary
A method for aggregating network flow information within a relational database relates to by maximizing concurrency between insertion and analysis of database records. In particular, data is generally stored according to the network devices associated with the flow records. Then, the flow records for the separate devices may be aggregated at certain time intervals and separately organized. In this way, contention is decreased as analysis can occur on the aggregated flow records, while new flow records are stored. In another embodiment, the aggregated data can be reaggregated again at a second, larger time interval.