Distributed Object Store Using Erasure Coding and Parallel Metadata Paths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face challenges in optimizing storage usage and reducing cross-datacenter network traffic costs while maintaining high availability and strong consistency, especially when dealing with large storage space consumption and relatively cold data with rare concurrent updates.
Innovation Solution
A distributed data object management system that implements a strongly consistent, versioned object store using consensus protocols like Paxos and Fast Paxos, optimizing metadata and data paths for write and read operations across global data centers, and employing erasure coding to efficiently store and retrieve data objects, allowing for parallel execution of operations and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data objects are stored across multiple global data centers with strong consistency requirements, then reliability and availability are improved, but cross-datacenter network traffic costs and latency increase
Solution Approach 1:
The system segments data objects into erasure-coded fragments and distributes them across multiple data centers. Each data center stores a portion of the encoded data, allowing the system to achieve strong consistency and high availability while reducing the amount of data that needs to be transmitted across data center boundaries compared to storing complete replicas.
Solution Approach 2:
The system changes the storage parameter from full replication to erasure coding, where data is encoded into n fragments from k original data fragments. This parameter change allows the system to store data across multiple data centers with improved reliability while optimizing the trade-off between storage overhead and network traffic costs.
2Reliability
If erasure coding is used to store data objects across data centers, then storage efficiency and durability are improved, but storage overhead and complexity increase
Solution Approach 1:
The system implements a universal data object management interface that handles both erasure-coded storage and metadata management through a unified API. This multi-functional approach allows the system to provide durable storage with erasure coding while maintaining a simple interface for users, effectively managing the complexity of erasure coding operations.
Solution Approach 2:
The system introduces a data object manager as an intermediary layer between the user interface and the underlying erasure-coded storage system. This mediator handles the complexity of erasure coding operations, fragment management, and recovery processes, while presenting a simple storage interface to users, thereby managing storage overhead and operational complexity.
3Reliability
If metadata and data operations are performed sequentially, then consistency is maintained, but operation latency increases
Solution Approach 1:
The system performs preliminary actions by pre-computing erasure-coded fragments and pre-positioning metadata information before actual data operations are needed. This preliminary preparation allows the system to maintain consistency while reducing operation latency, as the heavy computational work is done in advance rather than during data access operations.
Solution Approach 2:
The system transitions from sequential single-dimensional processing to parallel multi-dimensional processing by executing metadata operations and data operations in parallel across different dimensions of the storage system. This dimensional change allows consistency to be maintained through coordinated parallel execution while significantly reducing overall operation latency.
Data Source
AI summary
In various embodiments, methods and systems for implementing distributed data object management are provided. The distributed data object management system includes a distributed storage system having a local metadata-consensus information store in and one or more remote metadata-consensus information stores. A metadata-consensus information store is configured to store metadata-consensus information. The metadata-consensus information corresponds to erasure coded fragments of a data object and instruct on how to manage the erasure coded fragments. The distributed storage system further includes a local data store and one or more remote data stores for the erasure coded fragments. The distributed data object management system includes a distributed data object manager for operations including, interface operations, configuration operations, write operations, read operations, delete operations, garbage collection operations and failure recovery operations. The distributed data object management system is operates based on metadata paths and data paths, operating in parallel, for write operations and read operations.