In-Memory Event Aggregation for Security Information Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Security information/event management systems face significant processing loads and delays due to the large volume of event data, particularly when generating reports, as existing systems read and process the same data multiple times for different queries, leading to inefficiencies and increased disk access.
Innovation Solution
Implementing in-memory processing and aggregation of trend results, where unique elements of queries are identified and computed once, reducing redundant evaluations and disk access, and pre-computing data to amortize the cost of running reports over a longer duration, thereby reducing the load on the security system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If events are stored on disk for future use and processed when needed, then data can be retained for historical analysis, but processing the same data multiple times for different queries creates significant load and delays
Solution Approach 1:
The system pre-computes and stores aggregated trend results in memory before they are needed for reporting. When events arrive, the system continuously aggregates them into trend data and maintains this pre-computed data in memory, so that when reports are requested, the data is already prepared and ready for immediate retrieval, eliminating the need to re-process raw events at report generation time.
Solution Approach 2:
The system buffers aggregated trend results in memory as a cushion between raw event data and final reports. This in-memory buffer absorbs the processing load by maintaining pre-aggregated data, cushioning the system against the impact of multiple concurrent report queries that would otherwise require repeated disk access and processing of the same event data.
2Adaptability or versatility
If the same event data is read and processed multiple times for different queries, then comprehensive reporting can be generated, but redundant processing increases system load and processing time
Solution Approach 1:
The system merges multiple query requirements into a single aggregation process. Instead of processing events separately for each query, the system combines the aggregation logic for multiple trends and queries into unified in-memory data structures that can serve multiple reporting needs simultaneously, reducing redundant processing while maintaining reporting flexibility.
Solution Approach 2:
The system performs preliminary aggregation of events into trend data in memory, organizing the data in a way that can efficiently serve multiple different queries. This pre-organization of data allows the system to respond to various reporting requirements without re-processing the same event streams, reducing processing time while maintaining adaptability.
3Reliability
If disk access is used for storing and retrieving event data, then data persistence is achieved, but frequent disk access slows down processing and report generation
Solution Approach 1:
The system loads aggregated trend results from disk into memory in advance and maintains them there during the aggregation period. This preliminary loading of data into the faster memory medium allows rapid access and processing without repeated disk I/O operations, while the system periodically persists updated aggregated data back to disk for long-term persistence.
Solution Approach 2:
The system introduces an in-memory buffer as an intermediary between disk storage and report generation. This memory buffer acts as a fast-access cache that holds recently aggregated trend data, mediating between the slow disk storage and the fast report generation process, thereby eliminating the need for frequent direct disk access while maintaining data persistence through periodic writes.
4Productivity
If aggregated trend results are maintained in memory, then processing speed improves, but memory resource consumption increases
Solution Approach 1:
The system maintains aggregated trend results in memory only for the duration needed to serve reporting queries, rather than permanently storing all raw event data in memory. Once the aggregation period expires or memory capacity is reached, the system allows the oldest aggregated data to expire or be overwritten, maintaining only the necessary portion of data in memory to support current reporting needs.
Solution Approach 2:
The system dynamically manages memory allocation for aggregated trend data, adjusting the amount of data retained in memory based on current system conditions, query patterns, and available resources. The aggregation windows and memory retention periods can be adjusted dynamically to optimize the balance between processing speed and memory consumption based on changing operational requirements.
Data Source
AI summary
Systems and methods for in-memory processing of events are provided. A set of unique elements of a plurality of queries is determined. Each query is executed on a defined schedule and time duration. A plurality of events in an event stream are received. The events are filtered using the set of unique elements. For each query, a query result for each filtered event is determined. For each query, in-memory aggregation of the query result of each filtered event is provided.