In-Memory Security Analytics With Parallel Rule Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security analytics platforms face increased latency and computational complexity due to the storage and processing of event data on non-volatile memory, which becomes significant as the number of rules and data throughput increase.
Innovation Solution
Implementing an in-memory rule evaluator within a security analytics platform that maintains a set of security rules in volatile memory, allowing each node to apply its subset of rules to incoming data without persisting it to disk, and adding nodes for parallel processing as needed to balance workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If event data is stored and processed on non-volatile memory, then data persistence and reliability are improved, but processing latency and computational complexity increase
Solution Approach 1:
The patent segments the security rules into multiple subsets and distributes them across different nodes in the platform. Each node processes a specific subset of rules in parallel, eliminating the need to read from non-volatile memory sequentially. This segmentation allows the system to maintain reliability through distributed processing while dramatically reducing latency by performing operations in parallel rather than sequentially on persistent storage.
Solution Approach 2:
The patent implements preliminary action by pre-loading security rules subsets into volatile memory of multiple nodes before processing begins. This allows the system to have data ready in fast memory ahead of time, eliminating the need to read from non-volatile memory during actual event processing. The rules are prepared and positioned in advance in the fastest available memory, reducing processing latency while maintaining complete rule coverage across nodes.
2Reliability
If the number of security rules and data throughput increase, then detection capability and security coverage are improved, but processing speed and system performance deteriorate
Solution Approach 1:
The patent divides the complete set of security rules into multiple subsets and assigns each subset to a different node. This segmentation allows the system to scale horizontally - as more rules are added to improve security coverage, new nodes can be added with additional rule subsets rather than overloading a single node. Each node processes its subset in parallel, maintaining processing speed even as total rule count and security coverage increase.
Solution Approach 2:
The patent transitions from a single-node sequential processing model to a multi-node parallel processing architecture. By adding the dimension of multiple processing nodes working simultaneously, the system can handle increased data throughput and rule complexity without sacrificing processing speed. The problem moves from one-dimensional sequential processing to multi-dimensional parallel processing, where throughput scales with the number of nodes.
3Measurement precision
If all security rules are applied to all incoming data, then detection accuracy is improved, but computational complexity and resource consumption increase
Solution Approach 1:
The patent segments the complete rule set into multiple subsets distributed across nodes. Each node applies its specific subset to all incoming data, maintaining comprehensive detection coverage. The segmentation reduces computational complexity at each individual node while preserving overall detection accuracy through the collective coverage of all nodes. No single node needs to handle the full rule set, distributing the computational burden.
Solution Approach 2:
The patent merges the results from multiple nodes, each applying different rule subsets, to achieve complete security rule coverage. While each node performs simpler individual computations on its subset, the combined output of all nodes provides the equivalent detection accuracy of a single node applying all rules. The complexity is distributed and then merged, maintaining accuracy while reducing per-node computational burden.
Data Source
AI summary
A method includes receiving, by a processing device of a security analytics platform, data associated with a computing resource and assigning a first subset of a set of security rules to a first node of the security analytics platform and a second subset of the set of security rules to a second node of the security analytics platform. The first node applies, to the data, the first subset of security rules to generate first analytics data and the second node applies, to the data, the second subset of security rules to generate second analytics data. The first analytics data and the second analytics data are sent to a system associated with the computing resource.