Network Traffic Metadata Grouping for Columnar Database Cardinality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for storing network traffic metadata in columnar databases result in sub-optimal distribution of cardinality across database partitions, leading to inefficiencies in query processing and analysis due to high cardinality of metadata.
Innovation Solution
A computer-implemented method for grouping network traffic metadata based on selected dimensions, distributing it into optimal groups to minimize cardinality across unselected dimensions, and storing these groups in different partitions of a columnar database, using statistics about traffic flow to ensure efficient data distribution and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If network traffic metadata is randomly partitioned and stored in columnar database partitions, then the storage structure is simple and easy to implement, but the cardinality of metadata in each partition becomes high which reduces query processing efficiency
Solution Approach 1:
The patent segments network traffic metadata into multiple partitions based on selected dimensions (e.g., source IP, destination IP, protocol) rather than random distribution. This segmentation reduces the cardinality of metadata in each partition by grouping related records together, thereby improving query processing efficiency while maintaining manageable partition structures
Solution Approach 2:
The patent applies local quality by optimizing each partition's data characteristics according to its specific dimension groupings. Each partition is designed to handle specific query patterns efficiently by having low cardinality for the partitioning dimension, while the system as a whole handles diverse query types through multiple partitioning strategies
2Productivity
If network traffic metadata is stored with high cardinality distribution, then all metadata records can be stored uniformly across partitions, but the analysis and query response efficiency deteriorates due to high cardinality
Solution Approach 1:
The patent segments metadata by selecting specific dimensions (such as source IP address, destination IP address, or protocol type) to partition the data. This segmentation reduces the effective cardinality within each partition by grouping metadata records that share common values in the partitioning dimension, thereby improving analysis efficiency without losing data uniformity
Solution Approach 2:
The patent changes the parameter of cardinality by transforming the distribution approach from uniform random partitioning to dimension-based partitioning. By changing how metadata is distributed across partitions (from random to structured), the effective cardinality that impacts query performance is reduced while maintaining comprehensive data coverage
Data Source
AI summary
A computer-implemented method of grouping network traffic metadata includes, based on a selected dimension of the network traffic metadata received from a network router, obtaining a statistic about a flow of network traffic metadata received over an interval for each instance of multiple instances of the dimension. The method further includes distributing the network traffic metadata into a plurality of groups for network traffic metadata from the smallest possible number of instances of the selected dimension to be distributed to each group, with the flow of network traffic metadata distributed optimally for a criteria regarding the statistic amongst the plurality of groups for minimizing cardinality of each group of the plurality of groups with respect to unselected dimensions of the network traffic metadata and providing each group to a columnar database for storage of the network traffic metadata distributed into each group in a different partition of the columnar database.