Erasure-Coded Object Store With Parallel Metadata Consensus
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face challenges in optimizing storage usage and reducing cross-datacenter network traffic costs while maintaining high availability and strong consistency, especially when dealing with large storage space consumption and relatively cold data with rare concurrent updates.
Innovation Solution
A distributed data object management system that implements a strongly consistent, versioned object store using erasure coding across global data centers, optimized with metadata and data paths for write and read operations, and employs consensus protocols like Paxos and Fast Paxos to achieve consistency and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple data centers to ensure high availability and strong consistency, then reliability is improved, but cross-datacenter network traffic cost increases
Solution Approach 1:
The patent segments data into erasure-coded fragments and distributes them across multiple data centers. Instead of replicating entire data objects, the system divides data into k data fragments and m parity fragments, storing them in different data centers. This segmentation reduces the amount of data that needs to be transmitted across data centers while maintaining reliability through the erasure coding mechanism.
Solution Approach 2:
The system changes the parameter of data representation from full replication to erasure-coded fragments. By transforming data into k+m fragments with specific redundancy properties, the system achieves the same reliability goals with reduced network traffic. The erasure coding parameters (k, m) can be adjusted to balance between reliability and network cost.
2Quantity of substance
If erasure coding is applied across global data centers to reduce storage overhead, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The patent introduces a metadata-consensus information store as an intermediary that manages the complexity of erasure coding operations. This metadata store keeps track of fragment locations, versions, and consistency information, abstracting the complex erasure coding logic from the data storage operations. The metadata layer mediates between the application layer and the distributed storage infrastructure.
Solution Approach 2:
The system creates a universal metadata-consensus information store that serves multiple functions: tracking fragment locations, managing versioning, coordinating consistency protocols, and enabling recovery operations. This multi-functional metadata infrastructure reduces overall system complexity by consolidating management responsibilities in a single component.
3Reliability
If consensus protocols are implemented to achieve strong consistency, then reliability is improved, but operation latency increases
Solution Approach 1:
The patent implements partial consensus actions by using context-aware consensus protocols that adapt the level of consensus required based on the operation type. For read operations, the system may use simplified consistency checks, while write operations trigger full consensus protocols. This partial application of consensus mechanisms reduces latency for common operations while maintaining strong consistency when required.
Solution Approach 2:
The system dynamically selects between different consensus protocols (Paxos, Fast Paxos) based on the operational context. The metadata-consensus information store adjusts the consensus mechanism being used according to the particular circumstances of each operation, enabling the system to optimize between consistency strength and latency requirements in real-time.
4Productivity
If parallel metadata and data operations are implemented to reduce latency, then productivity is improved, but system complexity increases
Solution Approach 1:
The patent segments operations into independent metadata operations and data operations that can execute in parallel. The metadata-consensus information store handles metadata updates independently from the actual data fragment operations, allowing these two critical paths to proceed concurrently without blocking each other, thus improving overall operation speed.
Solution Approach 2:
The metadata-consensus information store acts as an intermediary that coordinates parallel operations. It manages the synchronization and consistency of parallel metadata and data operations without requiring complex interlocking mechanisms, simplifying the coordination of concurrent operations while maintaining system integrity.
Data Source
AI summary
In various embodiments, methods and systems for implementing distributed data object management are provided. The distributed data object management system includes a distributed storage system having a local metadata-consensus information store in and one or more remote metadata-consensus information stores. A metadata-consensus information store is configured to store metadata-consensus information. The metadata-consensus information corresponds to erasure coded fragments of a data object and instruct on how to manage the erasure coded fragments. The distributed storage system further includes a local data store and one or more remote data stores for the erasure coded fragments. The distributed data object management system includes a distributed data object manager for operations including, interface operations, configuration operations, write operations, read operations, delete operations, garbage collection operations and failure recovery operations. The distributed data object management system is operates based on metadata paths and data paths, operating in parallel, for write operations and read operations.