Triggered Retention in Ordered Event Stream Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional event stream storage systems lack the capability to effectively manage retention policies, leading to the unintended deletion of critical events, as they rely on rudimentary methods such as deleting events older than a default date, which can result in the loss of important data, especially in scenarios where specific events need to be retained for investigation or analysis.
Innovation Solution
The implementation of a triggered retention window mechanism, where specific triggers, such as successful landing of an airplane or reaching a certain number of events, activate a retention period, ensuring that events are retained until all retention rules are satisfied, including both triggered and fixed retention policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If conventional rudimentary methods (deleting events older than a default date) are used to manage event stream storage, then storage space can be freed up, but critical events may be unintentionally deleted
Solution Approach 1:
The system performs preliminary actions by establishing retention policies and triggers in advance before events need to be deleted. Retention rules are defined beforehand with specific conditions (triggers) that must be met before deletion occurs, ensuring critical events are protected from premature deletion while allowing storage management.
Solution Approach 2:
The system implements feedback mechanisms by continuously monitoring event streams against defined retention policies. Triggers are evaluated based on event characteristics, and deletion decisions are made only when retention rules are satisfied, providing a feedback loop that prevents unintended deletion while optimizing storage space.
2Reliability
If triggered retention policies are implemented to prevent deletion of critical events, then data integrity is maintained, but storage management becomes more complex
Solution Approach 1:
The retention management system is segmented into distinct components: retention policies, triggers, and evaluation mechanisms. Each event can be independently evaluated against retention rules, allowing complex retention requirements to be broken down into manageable segments that are processed individually through the event stream.
Solution Approach 2:
The triggered retention system is designed as a universal framework that can handle multiple retention scenarios through a single mechanism. The same retention policy infrastructure manages diverse event types and retention requirements, reducing overall system complexity despite the sophisticated retention control capabilities.
Data Source
AI summary
Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events.