Fine-Grained Data Expiration in Streaming Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently managing storage capacity by deleting older data, as policies based on size or age often result in the loss of useful data or excessive retention of unnecessary data, leading to inefficient use of storage resources.
Innovation Solution
Implementing a fine-grained data expiration system that scans older stream segments, identifies unexpired events based on event information within the data, and copies these events to new segments, allowing for targeted retention and deletion of expired data, thereby reducing storage footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deletion policy is based on size or age to free storage capacity, then storage capacity is freed up, but useful data is lost
Solution Approach 1:
The patent applies local quality by implementing event-level expiration policies where different events within the same stream can have different expiration times based on their individual characteristics. This allows the system to retain useful events longer while expiring less important ones, thereby freeing storage capacity without losing valuable data. The expiration policy is applied locally to each event rather than uniformly to all data in a stream.
Solution Approach 2:
The patent changes the parameter of data expiration from coarse-grained (stream-level or segment-level) to fine-grained (event-level). By introducing event-specific expiration parameters that can be determined from event information such as logging levels, the system can dynamically adjust which events are retained and which are expired, resolving the contradiction between freeing storage and preserving useful data.
2Quantity of substance
If data deletion policy is based on age to free storage capacity, then storage capacity is freed up, but no longer needed data is retained
Solution Approach 1:
The patent applies local quality by implementing event-level expiration policies where different events within the same stream can have different expiration times based on their individual characteristics. This allows the system to retain useful events longer while expiring less important ones, thereby freeing storage capacity without losing valuable data. The expiration policy is applied locally to each event rather than uniformly to all data in a stream.
Solution Approach 2:
The patent changes the parameter of data expiration from coarse-grained (stream-level or segment-level) to fine-grained (event-level). By introducing event-specific expiration parameters that can be determined from event information such as logging levels, the system can dynamically adjust which events are retained and which are expired, resolving the contradiction between freeing storage and preserving useful data.
3Quantity of substance
If fine-grained event expiration is implemented by scanning and copying unexpired events, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the data stream into discrete events, each with its own expiration characteristics. The expiration process segments the stream into unexpired events (to be retained) and expired events (to be removed). This fine-grained segmentation enables precise control over data retention while managing complexity through systematic processing of individual events rather than bulk operations.
Solution Approach 2:
The patent uses copying as a core mechanism for implementing expiration. Unexpired events are copied from source segments to destination segments, while expired events are left behind and eventually removed. This copying approach simplifies the expiration process by focusing on what to retain rather than what to delete, and allows for efficient implementation using standard data copy operations.
Data Source
AI summary
The described technology is generally directed towards fine-grained data expiration to reduce storage capacity used in a streaming data storage system. Segments of events are scanned, with each event's payload evaluated to determine whether that event is expired or unexpired. For example, for a log event, an expiration period can be determined based on the event creation time and the event logging level, both of which are typically within the log event's payload. Unexpired events are copied to a new destination segment that replaces the original, scanned source segment; the expired events are deleted when the source segment is deleted, reclaiming storage capacity used by the expired events. Also described is maintaining and using segment and chunk metadata to bypass individual event processing when the metadata indicates that the chunk contains only expired events and can be safely deleted.