Ordered Event Stream Retention via Checkpoint Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional event stream storage systems face issues with data retention, as older events are often deleted prematurely, leading to potential data loss due to reboot or processing failures, and existing semantic schemes do not effectively ensure that events are read and processed reliably.
Innovation Solution
The system employs an 'at-least-once' semantic scheme with checkpoint mechanisms to ensure events are read and processed before deletion, using aggregated checkpoints to determine the earliest progress point and prevent premature expiration, allowing events to be retained until they are assuredly read, and utilizing scalable storage schemes to manage event distribution across segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If events are deleted from the event stream to free storage space, then storage capacity is improved, but data loss risk increases when readers or processes fail to read events before deletion
Solution Approach 1:
The system performs preliminary actions by establishing retention policies and checkpoints before events are deleted. Readers register their intent to read events, and the system proactively retains events for registered readers even before actual reading occurs, preventing data loss while enabling storage management for events not targeted by any reader
Solution Approach 2:
The system implements feedback mechanisms where readers register their reading intentions with the event stream storage system. The system continuously monitors which events are retained based on reader registrations and adjusts retention accordingly, creating a closed-loop control system that balances storage capacity with data retention reliability
2Loss of energy
If conventional deletion policies are used to manage event stream storage, then storage space is recovered, but events may be deleted before being read by processes
Solution Approach 1:
The system introduces an intermediary retention policy layer between the event stream storage and deletion processes. This intermediary mechanism tracks reader registrations and maintains events for registered readers, acting as a buffer that prevents direct deletion of potentially needed events while still enabling storage management for events without registered readers
3Reliability
If events are retained indefinitely to prevent data loss, then data retention reliability is improved, but storage capacity is consumed
Solution Approach 1:
The system applies local quality by implementing differentiated retention policies: events registered with readers receive extended retention to ensure reliable delivery, while events without registered readers follow standard deletion policies. This localized approach to retention ensures data reliability for needed events while preserving storage capacity overall
Data Source
AI summary
Retention of events of an ordered event stream is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events are added to a head of an OES segment, and pruning events from a tail of the OES segment can be valuable. Processing applications can register a processing scheme for a segment, e.g., at-least-once processing, exactly-once processing, etc., and can generate checkpoints indicating a degree of advancement in processing events of the segment. The ordered event stream can determine a cut point indicative of a progress point, that before which, events of an OES can be marked as ready for expiration. However, events that are marked for expiration can be retained to allow processing based on a checkpoint, e.g., expiration of the event can be refused until there is an assurance the event was read by the processing application.