Hybrid Distributed Storage System for Concurrent Failure Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed object storage systems face inefficiencies in storage overhead and input/output operations, particularly when dealing with concurrent failures of storage elements, as they either incur high storage capacity overhead or suffer from increased input/output operations due to the high number of fragments spread across multiple storage elements.
Innovation Solution
A distributed object storage system that employs a hybrid encoding method, utilizing a spreading module to select level-1 and level-2 fragment storage subsets and a hybrid encoding module to generate and store fragments, allowing for efficient storage and retrieval while tolerating multiple concurrent failures without the need for synchronization between storage elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to achieve redundancy, then reliability is improved, but storage capacity overhead increases significantly
Solution Approach 1:
The data object is divided into multiple fragments through erasure encoding, which are then distributed across storage elements. This segmentation allows the system to achieve redundancy without storing complete replicas of the entire data object, thereby reducing storage overhead while maintaining reliability.
Solution Approach 2:
The system changes the redundancy parameter from traditional replication (storing complete copies) to erasure coding (storing encoded fragments). This parameter change enables the system to tolerate concurrent failures with significantly lower storage overhead, as the encoded fragments can be reconstructed to recover the original data object even when some storage elements fail.
2Reliability
If a high number of fragments are spread across multiple storage elements, then concurrent failure tolerance is improved, but input/output operations increase
Solution Approach 1:
The system performs preliminary encoding of the data object into fragments before distribution. This preliminary action organizes the data in advance, allowing the system to tolerate failures without requiring complex real-time operations during failure events, thereby reducing the I/O burden during normal operations and failure recovery.
Solution Approach 2:
The system extracts only the necessary number of fragments needed for reconstruction, rather than requiring access to all distributed fragments. This extraction principle allows the system to tolerate concurrent failures by only retrieving the minimum required fragments for data recovery, thereby minimizing I/O operations during failure scenarios.
3Quantity of substance
If RAID systems are used for distributed storage, then storage capacity overhead is reduced, but synchronization complexity and rebuild time increase
Solution Approach 1:
The system uses erasure coding to generate encoded copies of data fragments rather than requiring exact replicas. This copying approach through encoding allows the system to achieve redundancy with less storage overhead than traditional RAID while avoiding the synchronization complexity, as each fragment can be independently encoded and distributed without requiring coordinated updates across all storage elements.
Data Source
AI summary
There is provided a distributed object storage system that includes several performance optimizations with respect to efficiently storing data objects when coping with a desired concurrent failure tolerance of concurrent failures of storage elements which is greater than two and with respect to optimizing encoding/decoding overhead and the number of input and output operations at the level of the storage elements.