Bloom Filter Index for Event Data Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional security platforms experience inefficiencies in event data retrieval due to the need for multiple disk I/O operations and CPU processing, as they fetch and filter large amounts of unnecessary data, leading to slow query processing and increased latency in distributed networks.
Innovation Solution
The implementation of a Bloom filter index to check for the existence of non-indexed database parameters before reading event data, reducing unnecessary data retrieval and processing by providing a first level of data culling, thereby minimizing disk I/O and CPU usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the data manager component retrieves all event data from the IPDB based on indexed parameters only, then complete event data is obtained, but unnecessary data increases disk I/O operations and processing time
Solution Approach 1:
The Bloom filter index performs preliminary filtering of event data before the main retrieval process. By checking if event data matches the query criteria using the Bloom filter structure, the system identifies and excludes non-matching records early, avoiding unnecessary disk I/O operations and CPU processing time while ensuring all matching records are eventually retrieved.
2Reliability
If the data manager component retrieves all event data from the IPDB based on indexed parameters only, then complete event data is obtained, but the number of disk I/O operations increases
Solution Approach 1:
The Bloom filter index performs preliminary filtering of event data before the main retrieval process. By checking if event data matches the query criteria using the Bloom filter structure, the system identifies and excludes non-matching records early, avoiding unnecessary disk I/O operations and CPU processing time while ensuring all matching records are eventually retrieved.
3Adaptability or versatility
If additional non-indexed variables are included in the query, then more specific filtering is achieved, but the data manager component must ignore these variables and retrieve all data anyway
Solution Approach 1:
The Bloom filter index structure is designed to handle both indexed and non-indexed query parameters universally. It can filter event data based on any variable type (device type, username, custom fields) without requiring separate indexing strategies, thereby maintaining query flexibility while improving processing efficiency through preliminary filtering.
4Reliability
If large amounts of event data are exchanged among multiple security platforms in a distributed network, then comprehensive query results are achieved, but network latency increases
Solution Approach 1:
The Bloom filter index extracts and filters out non-matching event data before it is exchanged across the network between security platforms. By performing preliminary filtering locally at each platform using the Bloom filter structure, only relevant matching records are transmitted across the network, reducing network latency and bandwidth consumption while maintaining comprehensive query results.
Data Source
AI summary
Embodiments of the invention relate to a method for retrieving event data. The method includes receiving, by an event management device, an event query requesting event data corresponding to a filtering parameter, the filtering parameter being a non-indexed database parameter of an Internet Protocol database (IPDB). The method includes generating, by the event manager device, a Bloom filter value based upon the filtering parameter included in the event query. The method includes comparing, by the event manager device, the Bloom filter value with a Bloom filter index entry of an index file of the IPDB, the index file corresponding to the requested event data, the Bloom filter index entry indicating existence of the filtering parameter as part of the IPDB and reading, by the event manager device, the entry from the index file database when the Bloom filter value matches the Bloom filter index entry.