Network Flow Data Aggregation in Relational Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network monitoring methodologies face challenges with large volumes of network flow information, leading to storage and processing capacity issues, as well as contention for accessing stored data, where sampling techniques do not adequately address the problem of data volume and may not provide a complete picture of network traffic.
Innovation Solution
A method for aggregating network flow information within a relational database by organizing data at specific time intervals, minimizing the number of database objects required, allowing analysis on aggregated flow records while new records are stored, and optionally re-aggregating data at larger intervals, using a system with raw and aggregated data tables, and an archival storage system for compressed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If network flow information is stored in a relational database, then data can be accessed for analysis, but storage capacity is quickly exceeded due to voluminous data
Solution Approach 1:
The patent segments the database into multiple tables organized by time intervals (e.g., hourly, daily, weekly tables). This segmentation allows the system to manage large volumes of network flow data by dividing it into manageable chunks, enabling efficient storage and retrieval without overwhelming the database capacity.
Solution Approach 2:
The patent implements periodic data retention and archiving strategies where flow records are retained in detailed form for specific time periods and then archived or summarized. This periodic action manages storage capacity by automatically transitioning data from active to archived status based on time-based criteria.
2Measurement precision
If network flow data is accessed for analysis, then meaningful results can be produced, but new network flow information cannot be stored simultaneously
Solution Approach 1:
The patent creates separate database tables for different time periods (e.g., current period table, historical period tables). This segmentation allows concurrent operations where analysis queries can access historical tables while new data continues to be inserted into the current period table without blocking either operation.
Solution Approach 2:
The patent introduces aggregated/summarized data tables as intermediaries between raw flow records and analysis queries. These intermediate tables contain pre-processed data that can be queried for analysis without requiring access to the full volume of detailed records, thereby reducing contention on the raw data tables during write operations.
3Quantity of substance
If sampling techniques are used to decrease data flow volume, then storage capacity is reduced, but a complete picture of network traffic is not provided
Solution Approach 1:
The patent applies different data retention qualities to different time periods and data types. Recent data is retained in full detail for complete analysis, while historical data is archived or summarized. This local quality approach ensures that the most relevant data maintains completeness while older data is managed efficiently.
Solution Approach 2:
The patent implements partial aggregation where only certain fields or certain time periods are summarized while others remain in detail. This allows the system to reduce overall data volume through selective aggregation rather than uniform sampling, preserving the ability to reconstruct complete traffic pictures when needed.
Data Source
AI summary
A method for organizing network flow information within a relational database relates to minimizing contention for the network flow information. In particular, data is generally aggregated at certain time intervals and separately organized. In this way, contention is decreased as analysis can occur on the separated existing flow records, which are optionally aggregated, while new raw flow records are stored. In another embodiment, the aggregated data can be re-aggregated at second, larger time intervals.