Watermarking Ordered Event Stream Commitment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional event stream storage systems face challenges in accurately reading events due to erroneous timestamping, leading to uncertainty about the completeness of returned events, especially when events are partially or not yet fully committed.
Innovation Solution
The introduction of watermarking based on progress metrics, including effective progress, reported progress, and raw progress, to ensure that only fully committed events are read, preventing the return of uncommitted or partially written data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional timestamping is used to facilitate reading events in a time window, then reading efficiency is improved, but event accuracy deteriorates due to erroneous reading of uncommitted events
Solution Approach 1:
The patent introduces watermark values as an intermediary mechanism between timestamping and event reading. The watermark represents the progress of committed events and acts as a mediator that ensures only fully committed events are returned, while still allowing efficient time-window-based queries. The watermark is computed based on the minimum commit time across all partitions and serves as a safe boundary for reading operations.
Solution Approach 2:
The system implements feedback by continuously tracking the watermark value based on commit progress and using it to control reading operations. The watermark is updated as events are committed across partitions, and this feedback information is used to dynamically adjust the safe reading boundary, ensuring accuracy while maintaining efficiency.
2Speed
If events are read based on timestamp without commitment verification, then reading speed is improved, but data completeness deteriorates due to inclusion of uncommitted events
Solution Approach 1:
The patent applies preliminary action by computing and maintaining the watermark value in advance based on the commit status of events across all partitions. This pre-computed watermark serves as a predetermined safe boundary that guarantees data completeness. Readers can immediately use this pre-established boundary without performing additional verification, thus maintaining high reading speed while ensuring no uncommitted events are returned.
3Reliability
If watermarking based on commit progress is implemented to ensure event accuracy, then event reliability is improved, but system complexity increases due to multiple progress metrics
Solution Approach 1:
The patent segments the event stream into multiple partitions and computes the watermark for each partition independently based on its commit progress. The overall watermark is then derived as the minimum across all partition watermarks. This segmentation approach allows the system to handle complexity in a modular fashion, where each partition can be managed independently, and the combination rule (minimum) is simple and intuitive.
Data Source
AI summary
Watermarking of an ordered event stream (OES) is disclosed. Writing events to an OES can advance populating of the OES, which can be correspond to progress values. Progress values can enable a reader to know where it is in an OES. A reported progress value can be determined from the progress values based on an event being committed to the OES. Moreover, an effective progress value can be determined from the progress values and monitoring the OES for events with that defer being committed to the OES. A watermark for the OES can be determined from the effective progress value such that reading beyond the effective progress value is restricted. This can provide confidence that the reader has read out properly committed events and has not omitted any events having deferred commitment.