Segmented Streaming Data Storage Expiration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing streaming data storage systems face challenges in efficiently managing data expiration due to the need for a single expiration period for all events in a stream, leading to conservative data retention and inefficient storage resource utilization.
Innovation Solution
Implementing fine-grained, time-based data expiration at the event level, where each segment in a stream can have a unique expiration time determined by the maximum expiration time of its events, allowing for automatic deletion of expired data based on segment-specific expiration times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a single expiration period is applied to all events in a stream, then data retention policy compliance is simplified, but storage efficiency deteriorates due to conservative retention of events with shorter expiration periods
Solution Approach 1:
The stream is divided into multiple segments, each with its own expiration time derived from the maximum expiration period of events in that segment. This segmentation allows different expiration policies to be applied to different portions of the stream, improving storage efficiency while maintaining policy compliance.
Solution Approach 2:
Each segment is assigned a local expiration time based on the specific expiration periods of events within that segment, rather than applying a uniform expiration period across the entire stream. This local quality approach allows events with shorter expiration periods to be deleted promptly while retaining events with longer expiration periods.
2Loss of energy
If event-level expiration tracking is implemented, then storage efficiency is improved by deleting expired events promptly, but system complexity increases due to maintaining separate expiration times for each event
Solution Approach 1:
Multiple events within a segment share a common expiration time, which is determined by the maximum expiration period among events in that segment. This merging reduces the number of expiration times that need to be tracked from individual event level to segment level, simplifying the system while maintaining storage efficiency.
Solution Approach 2:
The segment-level expiration time serves as a universal expiration marker for all events within that segment, eliminating the need for individual expiration tracking for each event. This multi-functionality approach simplifies the expiration management mechanism while achieving efficient storage resource utilization.
3Measurement precision
If segments are created with different expiration times, then precise control over data retention is achieved, but management complexity increases due to handling multiple expiration times across segments
Solution Approach 1:
The system dynamically determines segment expiration times based on the maximum expiration period of events within each segment. This dynamic approach allows the expiration times to adapt to the actual data characteristics while maintaining a simple management mechanism through automatic calculation.
Solution Approach 2:
The system automatically calculates and assigns expiration times to segments based on the expiration periods of events within them, eliminating the need for manual configuration of multiple expiration times. This self-service mechanism reduces management complexity while achieving precise data retention control.
Data Source
AI summary
The described technology is generally directed towards fine-grained data event expiration in a streaming data storage system. An event to append is given an expiration period, and the expiration time for the events in a data stream or segment of a data stream is the largest expiration time among events in the data stream or segment. Different segments can have different expiration times for their events. In a segment comprising a group of events, a subgroup of expired events prior to a stream cut are deleted by an expiration task. For a subgroup of unexpired events prior to a stream cut, the expiration task retains (does not delete) the subgroup of events. If a scaling operation is performed on a segment, the new successor segment or segments inherit the largest expiration time of the predecessor segment or segments.