Event Stream Reconciliation Using Sentinel-Bounded Windows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Unbounded streams of events in event-driven architectures face challenges in ensuring data reliability, continuity, and integrity due to the lack of defined boundaries, making it difficult for consumers to verify event production, delivery, and data integrity.
Innovation Solution
A reconciliation framework that converts unbounded streams into bounded, self-reconcilable streams by injecting sentinels (boundary markers) into the stream, dividing it into units (windows and shards) with embedded reconciliation rules, allowing consumers to verify event delivery and integrity using producer-generated attributes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If an unbounded stream of events is used in event-driven architecture, then the system can handle continuous event flow without predefined limits, but it becomes difficult to verify event production, delivery, and data integrity
Solution Approach 1:
The patent segments the unbounded event stream into bounded windows and shards by injecting sentinels (boundary markers) that define start and end points. This segmentation allows consumers to verify event delivery within each bounded unit while maintaining the overall continuous flow capability of the unbounded stream.
2Reliability
If sentinels are injected into the event stream to create bounded units, then event verification and reconciliation become possible, but the system complexity increases
Solution Approach 1:
The sentinels carry embedded reconciliation rules and producer-generated attributes that enable consumers to autonomously verify event integrity without requiring complex external validation systems. The bounded units self-contain all necessary verification information, reducing overall system complexity.
3Reliability
If the stream is divided into bounded units with reconciliation rules, then data integrity can be verified, but processing overhead increases
Solution Approach 1:
The patent applies reconciliation rules selectively at sentinel boundaries rather than continuously across the entire stream. This partial action approach verifies data integrity at critical checkpoints while minimizing processing overhead during event transmission and initial reception.
4Ease of operation
If reconciliation attributes are embedded in each bounded unit, then consumers can validate events independently, but the data structure becomes more complex
Solution Approach 1:
Producer-generated attributes and reconciliation rules are embedded in sentinels during the event production phase, before consumption occurs. This preliminary action prepares all necessary validation information in advance, enabling consumers to independently verify events without requiring additional complex data structures or external validation data.
Data Source
AI summary
A computing system is configured to (i) inject, into an unbounded stream of events produced by the computing system, a window open sentinel that indicates one or more reconciliation rules for the window, (ii) inject, into the unbounded stream of events, a shard open sentinel that indicates one or more reconciliation rules for a shard, wherein the shard is within the window, (iii) transmit a set of events produced within the shard, (iv) execute the reconciliation rules for the shard to thereby generate one or more reconciliation attributes for the shard, (v) inject, into the unbounded stream of events, a shard close sentinel that indicates the reconciliation attributes for the shard, (vi) execute the reconciliation rules for the window to thereby generate one or more reconciliation attributes for the window, and (vii) inject, into the unbounded stream of events, a window close sentinel that indicates the reconciliation attributes for the window.