Event Query Rewriting via Progress Identifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The sheer volume of events stored in event logs makes it impractical to detect anomalous activity in a timely manner, as it burdens the primary system's CPU resources, potentially reducing the performance of primary processes.
Innovation Solution
An event analysis system is introduced to receive and analyze events from primary systems, allowing for the detection of anomalous activity without overburdening the primary system's resources. This system includes an event collector, a message queue, a security information and event management (SIEM) system, an event organizer, and a query analyzer that rewrites event queries to efficiently identify relevant events.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If events are stored in an event log for security monitoring, then security detection capability is improved, but system resource consumption increases and primary process performance deteriorates
Solution Approach 1:
The system divides the security monitoring function into separate components: the primary system generates events and stores them in an event log, while a secondary analysis system processes the event queries. This segmentation allows the primary system to focus on its core functions without being burdened by security analysis, thus maintaining primary process performance while enabling comprehensive security monitoring.
Solution Approach 2:
An event log acts as an intermediary between the primary system and the security analysis system. The event log stores events generated by the primary system and provides them to the analysis system upon request, eliminating the need for the primary system to directly process security queries and thereby preserving its performance.
2Reliability
If all events in the event log are queried to detect anomalous activity, then detection completeness is improved, but query response time increases
Solution Approach 1:
The system pre-processes and stores events in an event log with organized structures that facilitate efficient querying. Events are captured and stored in advance with relevant attributes indexed, so when a security query is issued, the analysis system can quickly retrieve relevant events without scanning the entire log, thus maintaining detection completeness while reducing query response time.
Solution Approach 2:
The system changes the parameter of event storage by organizing events in the event log with specific attributes and structures that optimize for query efficiency. This parameter change allows the analysis system to filter and retrieve events based on query criteria without examining every event in the log, thereby reducing response time while maintaining complete detection capability.
3Measurement precision
If the primary system processes security analysis directly, then analysis accuracy is improved, but system resource consumption increases
Solution Approach 1:
The system segments the processing workload by separating event generation (primary system) from event analysis (separate analysis system). The primary system maintains high accuracy in event generation and storage, while the analysis system performs the computationally intensive security analysis, distributing the resource burden and preventing CPU overload on the primary system.
Solution Approach 2:
The event log serves as an intermediary that preserves analysis accuracy by maintaining a complete and accurate record of events, while allowing the analysis system to perform resource-intensive processing without consuming the primary system's CPU resources. The event log ensures data integrity and accuracy while the analysis system handles the computational workload.
Data Source
AI summary
Events from one or more primary systems associated with one or more tenants are received. The received events are stored in a message queue. At least a portion of the events in the message queue are ingested for organization and storage in a data store. One or more progress identifiers associated with ingesting of the events in the message queue are tracked. An event query is received from an external system. The event query is rewritten into a first component query for the data store and a second component query for the message queue based at least in part on a progress identifier. A result of the first component query and a result of the second component query are combined to determine a result of the event query.