Distributed Event Streaming with Monitored Guaranteed Delivery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing event streaming platforms lack resilience against cluster-wide failures, leading to data loss and operational disruptions, and multi-cluster deployments increase complexity without ensuring guaranteed delivery.
Innovation Solution
A distributed event platform with independent, non-replicating streaming modules hosted on isolated server clusters, coupled with a monitoring module that tracks event delivery and operational performance, ensuring high availability and guaranteed delivery by re-propagating events as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If event streaming platforms use replicated streaming modules across clusters, then reliability is improved, but device complexity increases and cluster-wide failures can still propagate
Solution Approach 1:
The system segments the event streaming architecture into independent server clusters, where each cluster contains non-replicating streaming modules that process events independently. This segmentation isolates failures to individual clusters while maintaining overall system availability, and reduces complexity by eliminating the need for cross-cluster replication synchronization.
2Reliability
If event streaming platforms implement multi-cluster deployment, then reliability is improved, but device complexity increases
Solution Approach 1:
The system divides the event streaming platform into multiple independent server clusters that operate autonomously. Each cluster contains streaming modules that independently process events without requiring replication across clusters. This segmentation provides resilience against failures while keeping each cluster simple and manageable.
Solution Approach 2:
The system introduces a monitoring module as an intermediary that coordinates between independent server clusters. This mediator tracks event propagation across clusters and manages failover scenarios without requiring complex replication protocols, thereby maintaining reliability while controlling complexity.
3Device complexity
If event streaming platforms use independent non-replicating modules, then device complexity is reduced, but guaranteed delivery becomes difficult
Solution Approach 1:
The system implements a monitoring module that provides feedback on event propagation status across independent streaming modules. This feedback mechanism tracks whether events have been successfully delivered to consumer applications and triggers re-propagation if delivery fails, thereby guaranteeing delivery while maintaining the simplicity of independent non-replicating modules.
Solution Approach 2:
The monitoring module performs preliminary tracking of event propagation status before delivery is confirmed. By proactively monitoring event delivery and identifying failures early, the system can initiate re-propagation actions to ensure guaranteed delivery without requiring complex replication mechanisms in place beforehand.
4Reliability
If event streaming platforms implement comprehensive monitoring, then guaranteed delivery is achieved, but device complexity increases
Solution Approach 1:
The monitoring module serves multiple functions: it tracks event propagation status, detects delivery failures, initiates re-propagation actions, and provides system-wide visibility. This multi-functional approach achieves guaranteed delivery through comprehensive monitoring while avoiding the complexity of multiple separate specialized components.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention is directed to a system and a method for ensuring high availability and guarantee delivery of event streams to target consumer applications. The distributed event streaming platform of the present invention is provided with an event streaming layer comprising a plurality of independent and non-replicating event streaming clusters that are configured to process events received from a producer module. A monitoring module is provided to monitor the propagation and delivery of each propagated event based on event notifications received from an event producer module and a plurality of the consumer modules.