In-Memory Event Graph Queries for Low-Latency Threat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing digital security systems face inefficiencies in processing event data due to network latency, inefficient query execution, and resource wastage, particularly in detecting patterns of malicious behavior, which can delay threat detection and increase computational load.
Innovation Solution
A local event query host maintains an event graph in memory, adding events in real-time and scheduling query instances based on trigger events, allowing for efficient execution and rescheduling of queries based on actual event data, using partial query states to optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If queries are executed against a centralized repository of event data over a network, then comprehensive security analysis can be performed, but network latency delays threat detection
Solution Approach 1:
The system divides the centralized security analysis function into distributed components. Local security agents execute queries against local copies of event graphs, segmenting the monolithic centralized processing into autonomous distributed units that operate independently, thereby eliminating network latency while maintaining comprehensive analysis capability
Solution Approach 2:
Event graphs are pre-synchronized to local security agents before queries are executed. This preliminary action of distributing and caching event data locally allows immediate query execution without network round-trips, reducing detection delay while ensuring queries operate on up-to-date security event data
2Reliability
If all queries are executed against the event graph, then complete security pattern detection is achieved, but computational resources are wasted on unnecessary queries
Solution Approach 1:
The query execution system dynamically adapts query scheduling based on trigger events detected in the event stream. When specific security-relevant events occur, corresponding queries are activated; otherwise, queries are suspended or deferred. This dynamic approach ensures complete detection of security patterns while avoiding execution of queries unrelated to current security events, optimizing resource utilization
Solution Approach 2:
The system implements feedback mechanisms where query execution results and trigger event detections inform subsequent query scheduling decisions. When queries successfully detect security patterns or when specific trigger events occur, the system adjusts query execution timing and selection, ensuring comprehensive security monitoring while preventing redundant computational expenditure on already-satisfied or irrelevant queries
3Loss of information
If the event graph is updated with all incoming event data in real-time, then complete security event tracking is maintained, but processing overhead increases
Solution Approach 1:
Different portions of the event graph are maintained with different update frequencies and synchronization levels based on their security relevance. Frequently accessed or security-critical event data is updated in real-time with high fidelity, while less critical data uses asynchronous or batch updates. This local quality differentiation preserves complete security event tracking for critical data while reducing overall processing overhead through optimized update strategies for non-critical data
Data Source
AI summary
An event query host can include an event processor configured to process an event stream indicating events that occurred on a computing device. The event processor can add representations of events to an event graph. If an event added to the event graph is a trigger event associated with a query, the event processor can also add an instance of the query to a query queue. The query queue can be sorted based on scheduled execution times of query instances. At a scheduled execution time of a query instance in the query queue, a query manager of the event query host can execute the query instance and attempt to find a corresponding pattern of one or more events in the event graph.