Storage Manager Scheduling for Erasure Code Decoding Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in achieving high durability and efficiency, particularly in web-based applications and cloud computing, where storing large amounts of data requires balancing redundancy for durability with the need for low costs and efficient performance.
Innovation Solution
The implementation of an object-redundant storage system that uses erasure encoding schemes and a storage manager to refine transmission times by dynamically selecting the most responsive storage devices and pipelining the decoding process, allowing for efficient data retrieval even with partial shard loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple storage devices to ensure durability, then data reliability is improved, but storage cost and system complexity increase
Solution Approach 1:
The data object is divided into multiple shards that are distributed across different storage devices. Instead of replicating entire objects, the system segments data into manageable pieces that can be independently stored and retrieved, reducing the complexity of managing full replicas while maintaining durability through distributed storage.
Solution Approach 2:
The system uses erasure encoding to transform data into a different representation where only a subset of shards is needed for reconstruction. By changing the parameter of data representation from exact copies to encoded fragments, the system achieves durability with fewer storage resources and reduced complexity.
2Reliability
If all shards of a data object are retrieved before decoding, then data integrity is ensured, but retrieval time increases
Solution Approach 1:
The system pre-identifies which shards are necessary for decoding the data object based on erasure encoding properties. Instead of waiting to retrieve all shards, the system determines in advance the minimum set needed for reconstruction, allowing parallel retrieval of only those essential shards and significantly reducing retrieval time while maintaining integrity.
3Reliability
If the system waits for all storage devices to respond before decoding, then complete data is ensured, but latency increases
Solution Approach 1:
The system implements a feedback mechanism where storage devices report their response status and the coordinator tracks which shards have been received. Based on this feedback, the system dynamically determines when sufficient shards have been collected to proceed with decoding, allowing the process to continue as soon as the minimum requirement is met rather than waiting for all devices, thus improving speed while ensuring completeness.
Data Source
AI summary
A storage manager may be used to schedule requests for a data object stored in data sources of a storage system, such as an object-redundant storage system. The storage manager may iteratively request sets of corresponding blocks of the data object from storage devices of the storage system. As the corresponding blocks are received, the storage manager may store the corresponding blocks in an input buffer. In response to receiving at least a particular number of corresponding blocks, the storage manager may remove the corresponding blocks from the input buffer and decode the corresponding blocks into a block of the data object. In response to free space in the input buffer reaching a threshold amount, the storage manager may request subsequent sets of corresponding blocks of the data object such that at least some corresponding blocks are stored in the input buffer when the threshold amount is reached.