Forking Data Streams in Streaming Storage Platforms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently managing shared data streams after forking, where one application's truncation of older data can conflict with another application's need for that data, leading to coordination issues and data loss.
Innovation Solution
A streaming data storage system that creates new streams with logical copies of the original stream, allowing seamless stitching for reader applications and managing truncation requests to ensure no application can delete data still needed by another, without requiring data copying or complex stream management by applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the original data stream is sealed for sharing by multiple processes after forking, then data integrity is maintained, but coordination complexity increases and truncation conflicts occur
Solution Approach 1:
The patent segments the data stream into multiple independent stream instances (stream A, stream B, stream C) that share a common historical portion but have independent future portions. This segmentation allows each process to work with its own stream instance while maintaining data integrity through shared history, eliminating the need for complex coordination during truncation operations.
Solution Approach 2:
The patent introduces a stream fork mechanism that acts as an intermediary between the original process and the forked process. This intermediary creates logical copies of the stream that appear identical to reader applications but are independently manageable, allowing truncation operations to be performed without coordination conflicts while maintaining data integrity through the shared historical portion.
2Reliability
If the data stream is copied for the new process instance, then process isolation is achieved, but storage resources are consumed and data redundancy increases
Solution Approach 1:
The patent creates logical copies of the data stream through forking, where the forked stream shares the historical portion with the original stream but has independent future portions. This copying approach achieves process isolation without duplicating the entire data stream, thereby conserving storage resources while maintaining the necessary isolation between processes.
Solution Approach 2:
The patent merges the historical portion of the data stream into a shared segment that is referenced by both the original and forked streams. This combining approach eliminates redundancy by allowing multiple stream instances to share common historical data while maintaining independent future portions, thus reducing overall storage requirements.
3Productivity
If one application truncates older data from the shared stream, then storage space is freed, but data needed by other applications is lost
Solution Approach 1:
The patent segments the data stream into shared historical portions and independent future portions. When one application needs to truncate data, it can do so in its own stream instance without affecting other applications that share the historical portion, as each stream instance has independent truncation capabilities while maintaining shared access to necessary historical data.
Solution Approach 2:
The stream fork mechanism acts as an intermediary that manages data availability between multiple applications. It allows one application to truncate older data from its stream instance while maintaining data availability for other applications through the shared historical portion, thus resolving the conflict between storage efficiency and data availability.
Data Source
AI summary
The technology describes forking (dividing) a data stream into multiple data streams comprising appending a first new data stream and a second new data stream to the original data stream. After forking, the two new data streams have different sets of stream fragments, with one or more of the older stream fragments common to both sets. Data events before the forking operation are read from the original data stream, while data events appended after the forking operation are appended to and read from only one of the data streams, based on which stream each writer/reader is registered to. Also described is truncating the original data stream based on one stream identifier, while leaving the original data intact for another stream identifier. Only when no stream needs a portion of the original data (resulting from separate truncation operations) is the data portion deleted.