Event Stream Processing Failover via Event Block Object Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Event stream processing systems face challenges in ensuring seamless failover and minimizing data loss when an ESP device fails, leading to potential service interruptions and data loss, especially in mission-critical operations.
Innovation Solution
A system is implemented with a failover mechanism that includes multiple ESP devices, where one device acts as a backup, ensuring continuous operation by switching to a standby device without service interruption, using in-messaging and out-messaging network connectors to manage event block objects and maintain data consistency across devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single ESP device is used for real-time data processing, then the system simplicity is maintained, but the reliability and continuity of service are compromised when the device fails
Solution Approach 1:
The system is segmented into multiple ESP devices (primary and standby) that can independently handle event stream processing. Each device maintains separate event block objects and processing state, allowing the standby device to take over seamlessly when the primary device fails, thus improving reliability without requiring a complete system redesign
Solution Approach 2:
The system changes the operational state parameter of ESP devices between 'active' and 'standby' modes. The primary device operates in active mode while the standby device monitors and prepares to take over. This parameter change enables seamless failover by switching the operational state from single-active to standby-active when failure occurs
2Reliability
If event block objects are published immediately upon processing, then real-time data delivery is achieved, but data loss occurs during failover transitions
Solution Approach 1:
The standby device performs preliminary actions by maintaining a ready state with pre-loaded event processing capabilities and monitoring the primary device's event block objects. This preliminary preparation ensures that when failover occurs, the standby device can immediately resume processing without significant time loss or data loss
Solution Approach 2:
The standby device creates and maintains copies of event block objects and processing state from the primary device. These copies are kept synchronized and ready, allowing the standby to take over seamlessly during failover. The copying mechanism ensures data completeness while minimizing transition time
3Reliability
If multiple ESP devices are maintained for failover capability, then service continuity is improved, but the complexity of managing device states and event blocks increases
Solution Approach 1:
The system implements dynamic state management where ESP devices can transition between 'active' and 'standby' states based on operational needs and failure conditions. The primary device dynamically publishes event blocks while the standby dynamically monitors and prepares to take over, creating a flexible and adaptive system that manages complexity through defined state transitions
4Productivity
If the standby device waits for primary device failure before taking over, then resource utilization is optimized, but service interruption occurs during the failure detection and switching process
Solution Approach 1:
The standby device continuously receives feedback from the primary device through event block publications and status updates. This feedback mechanism allows the standby to detect primary device failures in real-time and trigger failover immediately, minimizing service interruption while maintaining optimized resource utilization through the active-standby configuration
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In a computing device supporting a failover in an event stream processing (ESP) system, an event block object is received. A first status of the computing device as active or standby is determined. When the first status is active, a second status of the computing device as newly active or not newly active is determined. Newly active is determined when the computing device is switched from a standby to an active status. When the second status is newly active, a last published event block object identifier that uniquely identifies a last published event block object is determined. A next event block object is selected from a non-transitory computer-readable medium accessible by the computing device. The next event block object has an event block object identifier that is greater than the determined last published event block object identifier. The selected next event block object is published to an out-messaging network device.