Scalable CEP Alerter for SIEM Event Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional complex event processing (CEP) systems face challenges in scalability and computational efficiency, particularly in security information and event management (SIEM) systems, due to exponential growth in event processing, inflexibility in rule creation, and high memory overhead, limiting the number of active rules and alerting capabilities.
Innovation Solution
The CEP alerter operates in stages, utilizing a de-duplicated list of events to streamline processing, separating event sorting from rule processing, and employing a state machine to efficiently detect and alert on predefined patterns within event streams, allowing for real-time analysis and reduced memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional CEP systems process all events from multiple sources, then complete event analysis is achieved, but computational overhead and memory requirements increase exponentially
Solution Approach 1:
The patent segments the event processing workflow into distinct stages: event collection, normalization, de-duplication, enrichment, and analysis. By dividing the monolithic processing pipeline into modular stages, each handling specific transformation tasks, the system reduces computational overhead at each step while maintaining complete event analysis capability.
Solution Approach 2:
The patent applies preliminary actions by performing event normalization, de-duplication, and enrichment before the main analysis phase. Events are pre-processed to remove duplicates, standardize formats, and add contextual information in advance, reducing the computational burden during real-time threat detection and analysis.
2Adaptability or versatility
If the number of active rules in CEP systems is increased, then alerting capabilities improve, but memory overhead increases limiting scalability
Solution Approach 1:
The patent extracts and removes duplicate events from the event stream before they enter the rule processing engine. By identifying and eliminating redundant events through de-duplication logic, the system reduces the number of events that need to be evaluated against active rules, thereby decreasing memory overhead while preserving complete alerting capabilities.
Solution Approach 2:
The patent discards duplicate events that provide no additional analytical value while recovering and retaining unique events for further processing. This selective filtering approach maintains the integrity of threat detection by preserving all meaningful events while eliminating redundant data that would otherwise consume excessive memory resources.
3Speed
If event processing is performed in real-time without pre-processing, then response speed is maintained, but computational efficiency decreases
Solution Approach 1:
The patent performs preliminary actions by pre-processing events through normalization, de-duplication, and enrichment in background threads before they reach the real-time analysis engine. This asynchronous pre-processing prepares events in advance without blocking the main event stream, maintaining response speed while significantly improving computational efficiency.
Solution Approach 2:
The patent introduces an intermediary event buffer and processing queue between event collection and real-time analysis. This intermediary layer allows asynchronous pre-processing operations to occur without blocking the real-time response path, enabling computational efficiency improvements while preserving the speed requirements of threat detection.
Data Source
AI summary
A methodology and related system operable to store a plurality of complex event processing (CEP) rules, the CEP rules being based on a plurality of events that are to be monitored. The CEP rules are pre-processed by, e.g., generating and storing a de-duplicated list of events from the plurality of events that are to be monitored. A received event from a received event stream is compared to events in the de-duplicated list of events and when a match between the received event (e.g., an event instance) and any one of the events in the de-duplicated list of events is detected, the received event (the event instance) is stored in an input repository. The plurality of CEP rules are then applied to the received event in the input repository, and any other previously stored events in the input repository.