Leader-Follower Event Stream Processing Cluster
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current event stream processing methods in SDN and IoT networks are cumbersome due to centralized architectures and lack dynamic stream subscription capabilities, leading to inefficiencies in handling and processing event streams across data, control, and management planes.
Innovation Solution
A cluster of nodes is arranged in a leader-follower relationship with a manager node that coordinates membership and leadership, allowing for efficient event stream processing by synchronizing processing across nodes and dynamically changing leadership based on performance criteria, thereby ensuring high availability and reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a centralized architecture with load balancer is used for event stream processing, then the system structure is simple, but the processing efficiency and availability deteriorate when nodes fail
Solution Approach 1:
The system segments the centralized event stream processing into multiple independent nodes, each capable of processing events. The cluster divides responsibilities between leader nodes (processing events) and follower nodes (standby), allowing the system to maintain functionality even when individual nodes fail, thus improving reliability without significantly increasing overall system complexity
Solution Approach 2:
The system implements dynamic role assignment where nodes can transition between leader and follower states based on performance criteria and failure conditions. This dynamic behavior allows the system to adapt to changing conditions and maintain high availability through automatic failover, resolving the contradiction between simple structure and reliable operation
2Productivity
If a load balancer is used to distribute event streams, then incoming streams can be separated into partitions, but the system cannot handle dynamic stream subscriptions
Solution Approach 1:
The system implements dynamic stream subscription by allowing follower nodes to subscribe to event streams in real-time. When a follower node needs to take over processing, it can dynamically subscribe to the relevant streams from the leader node, enabling flexible adaptation to changing processing requirements without rigid pre-configuration
Solution Approach 2:
The system uses feedback mechanisms where follower nodes monitor leader node performance and stream processing status. Based on this feedback, follower nodes can dynamically adjust their subscriptions and readiness state, enabling the system to adapt to changing conditions and maintain both productivity and versatility
3Reliability
If multiple nodes process event streams independently, then availability increases, but synchronization and coordination become complex
Solution Approach 1:
The system uses asymmetric node roles where leader nodes perform event processing while follower nodes perform standby and synchronization. This asymmetric division of labor simplifies the synchronization model compared to fully symmetric multi-node processing, as followers only need to track leader state changes rather than coordinate equally with all other nodes
Solution Approach 2:
Follower nodes maintain copies of the event stream processing state from leader nodes. This copying mechanism simplifies synchronization by allowing followers to passively replicate leader state through event subscription, avoiding complex real-time coordination protocols while maintaining availability through redundant copies
4Reliability
If leadership changes are implemented based on performance criteria, then system reliability improves, but the coordination overhead increases
Solution Approach 1:
Follower nodes perform preliminary actions by pre-subscribing to event streams and maintaining ready-to-process state before actually taking over leadership. This preliminary preparation reduces the coordination overhead and time loss during failover, as followers are already positioned to immediately assume processing responsibilities when leadership changes occur
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods are provided to process an event stream using nodes arranged as a cluster in which the nodes are in a leader-follower relationship with one node being the leader at any given time. Tracking of node membership and leadership status can be conducted in a manager node for a variety of applications. Each node of the cluster can register in the manager node and can receive the same set of event streams as the other nodes of the cluster. The leader node can process an event of an event stream with the follower nodes tracking the processing by the leader node, where the leader node can publish the processed event. Leadership can be changed from the leader node to another node of the cluster based on a performance criterion of the leader node. Additional systems and methods can be implemented in a variety of applications.