Low-Latency Watch Event Streaming With Dual-Path Ordering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing event-driven architectures in distributed systems face high latency and variability, particularly in cloud-native integrations like AWS Lambda functions and Kafka, which compromise timely event delivery and data integrity, especially under heavy load or cold starts.
Innovation Solution
A dual-path data streaming mechanism is implemented, combining a slow path for strong event ordering and durability with a fast path optimized for low latency, using a first-in first-out cache to manage and order events from both paths, ensuring minimal latency and high reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cloud-native integrations like AWS Lambda functions and Kafka are used for event processing, then durability and event logging are improved, but latency and variability increase
Solution Approach 1:
The system segments the event processing path into two distinct paths: a fast path for low-latency event delivery and a slow path for durable event logging. The fast path uses direct pub/sub messaging to deliver events immediately, while the slow path uses Kafka for reliable event logging. This segmentation allows the system to achieve both low latency and high durability by routing events through appropriate paths based on requirements.
2Device complexity
If a single path is used for event streaming, then system complexity is reduced, but either latency or durability must be compromised
Solution Approach 1:
The system divides the event streaming architecture into two parallel paths with distinct responsibilities. The fast path prioritizes speed using direct pub/sub, while the slow path prioritizes durability using Kafka. This segmentation resolves the contradiction by showing that increased complexity at the architectural level enables simultaneous optimization of both latency and durability, rather than forcing a choice between them.
3Reliability
If event ordering guarantees are enforced, then data integrity is improved, but event delivery speed decreases
Solution Approach 1:
The system segments the responsibility for event ordering between the two paths. The fast path delivers events immediately without strict ordering guarantees, maximizing speed. The slow path processes events through Kafka which provides ordering guarantees for durability-critical events. This segmentation allows the system to provide ordering guarantees where needed without sacrificing overall event delivery speed.
4Loss of time
If dual-path mechanism is implemented, then latency is reduced, but system complexity and coordination overhead increase
Solution Approach 1:
The system introduces an intermediary component that mediates between the fast and slow paths. This intermediary handles the coordination of events from both paths, reconciling duplicates and ordering events before delivery to clients. By centralizing this coordination logic in a dedicated intermediary component, the system reduces the complexity burden on individual path components and makes the dual-path mechanism more manageable.
Data Source
AI summary
Systems and methods provide low-latency, ordered delivery of database change events to a watch client. In response to a database (DB) create, update, or delete (CUD) request, the system writes the request to a database. The database usually provides a change data capture stream of events to a publish/subscribe (pub/sub) system. This results in higher change event delivery latency and is referred to as a slow path. The system leverages a dual write approach, where the system asynchronously writes a corresponding change event directly to a fast path pub/sub system. A watch server consumes change events from the slow and fast path pub/sub systems and merges and orders the change events using a first-in first-out cache. An ordered event stream can be transmitted to the watch client with lower latency. The ordered event stream includes one or more batches, and each batch comprises a most recent change event version for each object that was changed.