Synthetic Stream Data Retention in Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream-based data storage systems face challenges in managing unbounded data due to limited storage resources, requiring mechanisms like data expiration and truncation, which can lead to data loss and disrupt applications that cannot process 'holes' or missing events.
Innovation Solution
A data storage system that includes a hold processing component to implement holds on portions of data streams, generating synthetic streams to prevent alteration of events, allowing for data retention while allowing other events to be truncated, thus maintaining storage efficiency and application functionality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data expiration and truncation mechanisms are implemented to manage unbounded data streams, then storage resource utilization is improved, but data integrity and application functionality deteriorate due to data loss and missing events
Solution Approach 1:
The patent segments the data stream into multiple synthetic streams, each representing a specific hold period. This segmentation allows the system to retain specific portions of data indefinitely while allowing other portions to be truncated, thus maintaining data integrity for held events while improving storage resource utilization for non-held events.
Solution Approach 2:
The patent extracts held events from the main data stream and places them into separate synthetic streams. This extraction mechanism prevents held events from being truncated or altered while allowing the main stream to undergo normal expiration and truncation, resolving the contradiction between storage efficiency and data integrity.
2Quantity of substance
If data expiration and truncation are applied to unbounded data streams, then storage resource constraints are addressed, but application functionality deteriorates due to inability to process missing events
Solution Approach 1:
The patent introduces synthetic streams as intermediary structures between the main data stream and application processing. These synthetic streams act as mediators that preserve held events in their original form while allowing the main stream to be truncated, ensuring applications receive complete event sequences without holes or missing data.
Solution Approach 2:
The patent creates copies of held events in synthetic streams while allowing the original main stream to be truncated. This copying mechanism ensures that applications can access complete event histories from synthetic streams while the system maintains storage efficiency through truncation of non-held events in the main stream.
3Reliability
If holds are implemented on portions of data streams to prevent alteration, then data retention for legal or contractual purposes is improved, but storage resource usage increases and system complexity worsens
Solution Approach 1:
The patent segments held events into separate synthetic streams, organizing complexity in a structured manner. Each synthetic stream corresponds to a specific hold period, making the system's data retention policy transparent and manageable while preventing alteration of held events.
Solution Approach 2:
The synthetic stream mechanism serves multiple functions simultaneously: it prevents data alteration, enables indefinite retention for legal/contradictory purposes, and allows the main stream to undergo normal truncation. This multi-functionality reduces overall system complexity by consolidating multiple requirements into a single unified approach.
Data Source
AI summary
Stream retention in a data storage system is described herein. A method as described herein can include receiving, by a device operatively coupled to a processor, a hold for a portion of a data stream, wherein the data stream comprises respective events indexed by time, and wherein the portion of the data stream is bounded by a first event of the data stream that is associated with a first time and a second event of the data stream that is associated with a second time; generating, by the device, a synthetic stream comprising the portion of the data stream; and preventing, by the device, events of the data stream that are included in the synthetic stream from alteration.