Event Processing System with Analytics Matrix
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems (DBMS) and data stream management systems (DSMS) either support high-volume stateful event processing or real-time analytics with complex queries, but not both effectively, and often face scalability issues when handling large workloads.
Innovation Solution
The proposed solution involves separating the logic and state of event processing, using an Analytics Matrix (AM) for maintaining state, and implementing a scalable architecture with separate tiers for handling rules, continuous queries, and ad-hoc queries, allowing for scalable stream processing and real-time analytics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If DBMS are used to support real-time analytics with complex queries, then real-time analytics capability is improved, but stateful event processing capability deteriorates
Solution Approach 1:
The system is divided into separate functional layers: a stream processing layer for stateful event processing and a query processing layer for real-time analytics. This segmentation allows each layer to be optimized for its specific function, resolving the contradiction between the two capabilities.
Solution Approach 2:
An intermediary layer (Analytics Matrix) is introduced between the stream processing layer and the query processing layer. This intermediary maintains state information in a structured format that can be efficiently queried, enabling both stateful processing and real-time analytics without direct conflicts.
2Productivity
If DSMS are used to process high-volume event streams, then stateful event processing capability is improved, but real-time analytics capability deteriorates
Solution Approach 1:
The system separates stream processing functions from analytics functions into distinct layers, allowing DSMS-style high-volume event processing while adding real-time analytics capability through the query processing layer that accesses the Analytics Matrix.
3Productivity
If state variables are maintained for rule evaluation, then complex event processing capability is improved, but system scalability deteriorates
Solution Approach 1:
The Analytics Matrix serves as an intermediary that externalizes state variable storage from the stream processing logic. This allows state to be maintained for complex event processing while the structured format enables efficient scaling through optimized query processing on the matrix.
Solution Approach 2:
State variables are organized into the Analytics Matrix with specific dimensional structures (rows for state variables, columns for event attributes). This dimensional organization enables efficient access patterns that scale well with system size while maintaining complex processing capabilities.
4Device complexity
If data and workload are replicated and partitioned, then system scalability is improved, but real-time analytics performance deteriorates
Solution Approach 1:
The system segments state information into the Analytics Matrix with a specific structure that is optimized for query access. This structured segmentation allows scaling through partitioning while maintaining fast query performance by organizing data in query-friendly dimensions.
Data Source
AI summary
An event processing system is configured to process a stream of events operating on a database system. The event processing system comprises an event load balancing unit, a plurality of event computing nodes, and a plurality of event state stores, wherein the event load balancing unit is configured to route the stream of events to the plurality of event computing nodes, wherein the plurality of event state stores are configured to store states of the plurality of event computing nodes for maintaining a state of the event processing, and wherein the plurality of event computing nodes are configured to process the events, to change their states, and to update the plurality of event state stores based on their changed states.