Distributed Object Storage With Erasure Coding and Metadata Consensus
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face challenges in optimizing storage usage and reducing cross-datacenter network traffic costs while maintaining high availability and strong consistency, especially when dealing with large storage space consumption and relatively cold data with rare concurrent updates.
Innovation Solution
A distributed data object management system that implements a strongly consistent, versioned object store using erasure coding across global data centers, optimized with metadata and data paths for write and read operations, and employs consensus protocols like Paxos and Fast Paxos to achieve consistency and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple data centers to ensure high availability and strong consistency, then reliability is improved, but cross-datacenter network traffic cost increases
Solution Approach 1:
The patent segments data into erasure-coded fragments and distributes them across multiple data centers. Instead of replicating entire data objects, only necessary fragments are transmitted across data centers, reducing network traffic while maintaining availability and consistency through the erasure coding reconstruction capability.
Solution Approach 2:
The system changes the replication parameter from full data replication to erasure-coded fragment distribution. By using erasure coding with configurable redundancy levels, the system optimizes the balance between reliability and network traffic costs, allowing flexible adjustment based on specific workload requirements.
2Quantity of substance
If erasure coding is applied across global data centers to reduce storage overhead, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The patent introduces a distributed data object manager as an intermediary layer that handles erasure coding operations, metadata management, and coordination across data centers. This intermediary abstracts the complexity of erasure coding from the underlying storage system, making the implementation more manageable while achieving storage efficiency.
Solution Approach 2:
The system implements self-service mechanisms where the distributed data object manager automatically performs erasure coding, fragment distribution, and reconstruction operations without manual intervention. The metadata-consensus information stores enable automated tracking and management of erasure-coded fragments across data centers.
3Reliability
If consensus protocols are implemented to achieve strong consistency, then reliability is improved, but operation latency increases
Solution Approach 1:
The patent segments the consistency problem by applying consensus protocols selectively to metadata-consensus information rather than all data operations. This segmentation allows fast paths for operations that don't require full consensus, reducing latency while maintaining strong consistency where needed.
Solution Approach 2:
The system implements partial consensus action by using metadata-consensus information stores that track only the necessary consistency-critical metadata. This partial approach to consensus reduces the overhead compared to full consensus on all operations, achieving strong consistency for critical operations while allowing faster paths for others.
4Productivity
If parallel metadata and data paths are used for write and read operations to reduce latency, then productivity is improved, but system complexity increases
Solution Approach 1:
The patent segments the data management system into separate metadata-consensus information stores and data stores, with corresponding parallel metadata paths and data paths. This segmentation enables independent optimization of each path and simplifies the management of parallel operations by clearly defining the scope and interaction points between paths.
Solution Approach 2:
The distributed data object manager serves as an intermediary that coordinates between parallel metadata and data paths. It manages the interaction between metadata operations and data operations, ensuring consistency while enabling parallel execution, thus reducing latency without overwhelming complexity.
Data Source
AI summary
In various embodiments, methods and systems for implementing distributed data object management are provided. The distributed data object management system includes a local metadata-consensus information store and one or more remote metadata-consensus information stores for metadata-consensus information and a local data store and one or more remote data stores for erasure coded fragments. For a write operation, corresponding metadata writes and data writes are performed in parallel using a metadata write path and a data write path, respectively, when writing to the local metadata-consensus information store and the one or more remote metadata-consensus information stores and the local data store and the one or more remote data stores. And, for a read operation, corresponding metadata reads and data reads are performed in parallel using a metadata read path and a data read path, respectively, when reading from the metadata-consensus information stores and the data stores.