Merging Transaction Segments in Data Storage Without Epoch Transitions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream-based data storage systems face difficulties in efficiently merging transactions into streams when the transaction segments differ from the stream segments, leading to complications in scaling and resource management, particularly due to the need for epoch transitions which can block scale-up and scale-down events.
Innovation Solution
A data storage system that includes a merging component capable of merging transaction segments into stream segments without triggering epoch transitions, by leveraging epoch compatibility and key range matching, thereby reducing resource costs and preserving the stream structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If epoch transitions are performed to merge transaction segments into stream segments, then transactions can be committed to the stream, but the number of epoch transitions increases leading to blocked scale-up and scale-down events and increased resource overhead
Solution Approach 1:
The system performs preliminary actions by pre-aligning transaction segment key ranges with stream segment key ranges before committing transactions. This preparation ensures that transactions can be merged without triggering epoch transitions, thereby maintaining scaling efficiency while ensuring reliable transaction commits.
Solution Approach 2:
The system changes the parameter of key range alignment by adjusting transaction segment key ranges to match stream segment key ranges. This parameter adjustment allows transactions to be committed without epoch transitions, resolving the contradiction between transaction reliability and scaling efficiency.
2Productivity
If transaction segments are merged into stream segments without epoch transitions, then resource overhead is reduced and scaling is improved, but the distinct structure of transactions and streams must be reconciled
Solution Approach 1:
The system applies local quality by reconciling the structure of transaction segments and stream segments locally at the key range level rather than requiring global epoch transitions. This allows transactions to be merged into streams with matching key ranges without triggering system-wide structural changes, reducing resource overhead while managing structural complexity.
3Speed
If the number of epoch transitions is reduced, then system reactivity and performance are enhanced, but the requirement for key range matching between transaction and stream segments increases
Solution Approach 1:
The system performs preliminary key range alignment before transaction commits, ensuring that transaction segments are pre-matched with corresponding stream segments. This preliminary action reduces the need for epoch transitions during transaction processing, enhancing system reactivity while maintaining precise key range matching.
Data Source
AI summary
Efficient rolling transactions for a data storage system is described herein. A method as described herein can include structuring, by a device operatively coupled to a processor, a data stream according to a first structure, the first structure comprising respective stream segments that are associated with respective first key ranges; writing, by the device, a transaction to the data stream, wherein the transaction comprises events arranged in a second structure that is distinct from the first structure, and wherein the second structure comprises respective transaction segments that are associated with respective second key ranges; and merging, by the device, respective ones of the transaction segments into respective ones of the stream segments, resulting in merged transaction segments, in response to the second key ranges of the respective merged transaction segments at least partially matching the first key ranges of the respective ones of the stream segments.