Self-Indexed Searchable Object for Network Flow Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing flow log databases, such as Elastic and Lucene, face inefficiencies in storing and searching massive network traffic data due to their general-purpose nature, requiring massive storage and processing resources and lacking specialized indexing for rapid search capabilities.
Innovation Solution
A self-indexed searchable object system that uses a distributed object store with shards and index tables, allowing for efficient storage and retrieval of network traffic logs by indexing specific fields, enabling rapid search and reducing storage overhead through compression and append-only data handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If general purpose databases (Elastic, Lucene) are used for flow log databases, then the system can store network traffic data, but the storage and processing resources required become excessively large
Solution Approach 1:
The patent divides the flow log database into multiple shards distributed across different storage locations. Each shard contains a portion of the log entries, allowing parallel processing and reducing the burden on individual database nodes. This segmentation enables scalable storage while maintaining efficient query performance through distributed architecture.
Solution Approach 2:
The patent introduces an intermediary indexing layer that sits between the raw log entries and the query interface. This index structure pre-processes and organizes log data by relevant fields (source IP, destination IP, ports, protocols), enabling rapid lookup without scanning entire datasets. The intermediary index dramatically reduces processing time and resource consumption for common query patterns.
2Speed
If general purpose databases are used for flow log storage, then data can be stored, but search operations become slow due to lack of specialized indexing
Solution Approach 1:
The patent performs preliminary indexing actions during data ingestion, organizing log entries by multiple fields (source IP, destination IP, source port, destination port, protocol) before queries are executed. This pre-processing creates optimized index structures that enable rapid search operations. The indexing is performed once during data loading rather than during each query, dramatically improving search speed.
Solution Approach 2:
The patent creates a multi-functional index structure that serves multiple query types simultaneously. The same indexed data structure supports searches by source IP, destination IP, port ranges, and protocol types without requiring separate indexes for each query pattern. This universal indexing approach improves search performance across diverse query workloads while avoiding the complexity of maintaining multiple specialized indexes.
3Reliability
If flow log databases store records for every network packet, then complete traffic analysis is achieved, but the database size becomes extremely large
Solution Approach 1:
The patent extracts and stores only the essential flow log fields required for network traffic analysis (source IP, destination IP, source port, destination port, protocol, packet count, byte count, timestamps). Non-essential or redundant data is excluded from storage. This selective extraction maintains the completeness of traffic analysis capabilities while dramatically reducing the volume of stored data compared to storing every packet detail.
Solution Approach 2:
The patent implements partial aggregation of flow log data by combining multiple packets into single flow records. Instead of storing individual packet records, the system aggregates packets into flow entries that summarize traffic patterns (packet counts, byte counts, duration). This partial aggregation approach retains sufficient detail for comprehensive traffic analysis while reducing database size by orders of magnitude compared to storing every individual packet.
Data Source
AI summary
Network appliances can record log entries in log objects. An object store can receive the log objects and can use the log objects to create index objects and flow log objects. Each flow log object and index object can be associated with a time period wherein the flow log object includes flow log entries received during that time period. The index object includes shard tables that can be stored in different nonvolatile memories and can thereby be concurrently searched. Shard entries in the shard tables indicate flow entry indicators. The flow entry indicators indicate log entries in the flow log object. An internally indexed searchable object can include the flow log object and the index object. Numerous indexed fields in the flow log entries and can be indexed with each indexed field searchable via the shard entries.