Hierarchical Complex Event Processing System for Dynamic Data Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current complex event processing (CEP) systems are limited by assumptions about uniform data sources and static data structures, struggling to effectively process and detect complex events from diverse and dynamic data sources, which is crucial for timely threat detection and response in security and other applications.
Innovation Solution
The proposed system employs a hierarchical complex event processing (H-CEP) approach that uses templates with assertions and constraints to generate and process events, allowing for the integration of both active and passive data sources, and includes a constraint solver to filter event matches and produce viable solutions, enabling the detection of complex event patterns and triggering appropriate actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If current CEP systems are used with assumptions of uniform data sources and static data structures, then system simplicity is maintained, but the ability to process diverse and dynamic data sources deteriorates
Solution Approach 1:
The patent implements dynamic data source configurations that allow the CEP system to adapt to changing data structures and sources in real-time. The system uses runtime schema validation and dynamic query generation that can handle evolving data formats without requiring system redesign, thus improving adaptability while managing complexity through flexible architecture.
Solution Approach 2:
The patent creates a universal event processing framework that can handle multiple data source types (relational databases, NoSQL databases, message queues, files) through a common interface. The unified event model and standardized processing pipeline enable the system to work with diverse data sources using the same core logic, improving versatility without proportionally increasing complexity.
2Measurement precision
If complex event patterns are detected with multiple assertions and constraints, then detection accuracy is improved, but processing time increases
Solution Approach 1:
The patent implements preliminary indexing and pre-computation of constraint satisfaction results. The system pre-processes data to create indexes on frequently queried fields and pre-evaluates constraint relationships, so that during actual event detection, the system can quickly retrieve and combine pre-computed results rather than calculating everything from scratch, thus maintaining high detection accuracy while reducing processing time.
Solution Approach 2:
The patent divides complex event detection into segmented processing stages: event filtering, constraint validation, pattern matching, and result aggregation. Each stage handles a specific subset of the detection logic independently, allowing parallel processing and optimization at each level. This segmentation maintains comprehensive detection accuracy while enabling time-efficient processing through staged evaluation.
3Reliability
If comprehensive constraint solving is performed on all assertions, then solution reliability is improved, but computational resources are excessively consumed
Solution Approach 1:
The patent extracts and prioritizes critical constraints from the full set of assertions, identifying only the essential constraints that must be satisfied for a solution to be valid. The system separates mandatory constraints (which must be fully evaluated for reliability) from optional or less critical constraints (which can be evaluated selectively or with reduced precision), thus maintaining solution reliability for key requirements while reducing overall computational resource consumption.
Solution Approach 2:
The patent dynamically adjusts constraint evaluation parameters based on resource availability and event priority. For high-priority events, the system performs comprehensive constraint solving with full reliability checks. For lower-priority events or when resources are constrained, the system adjusts evaluation depth and precision parameters, performing only the most critical constraint checks. This parameter adaptation maintains reliability where needed while optimizing resource usage across the system.
Data Source
AI summary
Systems and methods for processing events are disclosed. Event data comprising passive event data, active event data, or both is received. It is determined whether the received event data is available for a pattern of passive event data and active event data. In response to determining that the received event data is available for the pattern of passive event data and active event data, one or more constraints between the passive event data and the active event data are converted into one or more query terms. The query terms are used to construct at least one query. Remaining passive event data that is related to some, but not all, of the active event data is obtained using the constructed at least one query.