Historic Netflow Analysis with Blob Indexes and Bitsets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Historical netflow data analysis is computationally expensive and storage-intensive due to the vast volume of data, making it impractical to search through millions of netflows efficiently.
Innovation Solution
A system and method utilizing blob index files and bitsets to summarize and granularize historical netflow data, reducing unnecessary data processing by generating blob index files and bitsets at various time intervals, enabling efficient and cost-effective search and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If historical netflow data is stored in complete detail for analysis, then analysis accuracy is improved, but storage requirements and processing time increase excessively
Solution Approach 1:
The patent extracts only the essential identifying information (IP addresses) from the complete netflow data and stores it in bitset structures. This allows the system to retain the ability to accurately identify and analyze historical netflows while dramatically reducing the storage requirements by eliminating redundant packet-level details.
Solution Approach 2:
Instead of storing complete netflow records and searching through them, the patent inverts the approach by creating compact bitset representations that can be quickly searched. The bitset structure allows the system to check for the presence of specific IP addresses in historical data without having to load or parse the original detailed netflow records.
2Measurement precision
If complete historical netflow data is searched, then detection accuracy is improved, but processing time becomes excessively long
Solution Approach 1:
The patent creates a simplified copy of the historical netflow data in the form of bitset structures. These bitsets contain the essential information needed for detection (presence of IP addresses) in a highly compressed format that can be processed and searched much faster than the original detailed netflow data, while maintaining detection accuracy.
Solution Approach 2:
The patent changes the data representation parameters by converting detailed netflow records into bitset format. This parameter change transforms the data from a verbose, time-consuming format into a compact, efficiently searchable format, dramatically reducing processing time while preserving the ability to accurately detect and analyze historical netflows.
3Productivity
If computational power is increased to handle large historical datasets, then analysis capability is improved, but cost becomes excessively prohibitive
Solution Approach 1:
The patent uses inexpensive bitset data structures as a disposable intermediate representation for historical netflow analysis. Instead of investing in expensive high-performance computing infrastructure to process raw netflow data, the system uses cheap bitset operations that can be performed efficiently on standard hardware, making the analysis capability affordable and scalable.
Data Source
AI summary
A system and method perform historic netflow searching for an IP address, such as an IPv4 address, in a cost, time and storage efficient manner using blob indexes and bitsets.