Network Flow Log Database Sharding and Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
General-purpose databases like Elastic and Lucene struggle with efficiently storing and searching massive network traffic flow logs due to their resource-intensive nature and inability to handle massive data volumes effectively.
Innovation Solution
A specialized data store system that uses an object store with self-indexed searchable objects, optimized for network traffic flow logs, which employs a flat file format for efficient storage and search operations, allowing for concurrent reading and writing, and utilizing Protobuf for serialization and compression to reduce CPU and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general-purpose databases like Elastic and Lucene are used to store flow logs, then the system can handle diverse data types, but the resource consumption (CPU and memory) increases significantly
Solution Approach 1:
The patent segments the flow log database into multiple sharded index files, where each shard contains a subset of the data. This segmentation allows parallel processing during search operations, distributing the computational load across multiple CPU cores and reducing overall resource consumption while maintaining the ability to handle diverse data types.
Solution Approach 2:
The patent creates simplified copies of the data in a columnar format that are optimized for specific search operations. Instead of storing all data in a complex general-purpose database structure, it creates streamlined representations that reduce memory overhead while preserving the essential data types needed for network flow log analysis.
2Adaptability or versatility
If general-purpose databases are used for flow log storage, then the system can provide flexible querying, but the search speed decreases due to resource overhead
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing aggregated statistics (such as min, max, and sum values) in each index shard. This allows the search system to quickly filter out irrelevant shards before performing detailed searches, significantly improving search speed while maintaining flexible querying capabilities through the columnar data structure.
Solution Approach 2:
By dividing the database into sharded index files that can be searched in parallel, the system achieves both fast search speeds and flexible querying. Each shard can be processed independently by different CPU threads, enabling scalable performance improvement as data grows while preserving query flexibility through the underlying data model.
3Quantity of substance
If massive volumes of network traffic flow logs are stored, then the database size increases, but the performance of general-purpose databases degrades
Solution Approach 1:
The patent implements sharding that divides massive volumes of flow log data into multiple manageable index files. Each shard maintains efficient search performance independently, allowing the system to scale to petabytes of data while preserving operational efficiency. The segmentation enables linear scalability where adding more shards proportionally increases both capacity and processing power.
Solution Approach 2:
The patent changes the fundamental parameters of data storage by using a columnar format with variable-length encoding and compression. This transforms the data structure from row-based to column-based organization, enabling efficient storage of massive data volumes while maintaining fast search performance through vectorized operations on columnar data.
4Speed
If a specialized format is used for flow log storage, then search efficiency improves, but the device complexity increases
Solution Approach 1:
The patent applies homogeneity by using a uniform columnar data structure across all index shards. Each shard follows the same format with consistent column definitions, encoding schemes, and compression methods. This homogeneous structure simplifies the complexity management by providing a predictable, repeatable pattern that can be processed efficiently by standardized search algorithms, despite the specialized nature of the format.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Network appliances can record log entries in log objects. An object store can receive the log objects and can use the log objects to create index objects and flow log objects. Each flow log object and index object can be associated with a time period wherein the flow log object includes flow log entries received during that time period. The index object includes shard tables that can be stored in different nonvolatile memories and can thereby be concurrently searched. Shard entries in the shard tables indicate flow entry indicators. The flow entry indicators indicate log entries in the flow log object. An internally indexed searchable object can include the flow log object and the index object. Numerous indexed fields in the flow log entries and can be indexed with each indexed field searchable via the shard entries.