Hybrid Distributed Storage System for Concurrent Failure Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face inefficiencies in storage overhead and processing power when dealing with concurrent failures, particularly in systems that require tolerance for more than two storage element failures, as well as high input/output operations, which can lead to bottlenecks in network bandwidth, storage capacity, and processing power.
Innovation Solution
A distributed storage system utilizing a hybrid encoding and retrieval method that combines fragment storage and replication copies, where a hybrid encoding module generates encoding multiples of fragments and replication copies, allowing for flexible redundancy and minimal processing power and input/output operations, enabling efficient storage and retrieval even under high loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to achieve redundancy for surviving storage element failures, then reliability is improved, but storage capacity overhead increases significantly
Solution Approach 1:
The data object is divided into multiple fragments using erasure encoding, which are then distributed across storage elements. This segmentation allows the system to tolerate concurrent failures more efficiently than full replication, reducing storage overhead while maintaining reliability.
Solution Approach 2:
The patent combines multiple redundancy schemes (erasure encoding fragments and replication copies) into a hybrid approach. This composite strategy leverages the strengths of both methods to achieve better concurrent failure tolerance with optimized storage capacity utilization.
2Quantity of substance
If erasure encoding is used to reduce storage overhead, then storage capacity efficiency is improved, but processing power requirements increase
Solution Approach 1:
Instead of fully decoding all fragments, the system retrieves only the necessary number of fragments or replication copies required to reconstruct the data object. This partial action reduces processing power requirements while maintaining storage efficiency benefits.
Solution Approach 2:
The system creates replication copies of the data object and stores them alongside erasure encoded fragments. This copying strategy provides redundant access paths that reduce the computational burden of erasure decoding, thereby lowering processing power requirements.
3Reliability
If high redundancy levels are implemented to tolerate more concurrent failures, then reliability is improved, but input/output operations increase causing bottlenecks
Solution Approach 1:
Replication copies provide direct access to the complete data object without requiring multiple I/O operations to assemble fragments. This copying mechanism reduces I/O overhead while maintaining high concurrent failure tolerance through the hybrid redundancy approach.
Solution Approach 2:
The system pre-distributes both erasure encoded fragments and replication copies across storage elements during the write operation. This preliminary action ensures that during read operations, the system can quickly retrieve data using the most efficient path available, reducing I/O bottlenecks.
Data Source
AI summary
There is provided a distributed object storage system that includes several performance optimizations with respect to efficiently storing data objects when coping with a desired concurrent failure tolerance of concurrent failures of storage elements which is greater than two and with respect to optimizing encoding/decoding overhead and the number of input and output operations at the level of the storage elements.