Peer-to-Peer Metadata Federation for Geo-Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise datacenters face scale-up and scale-out challenges due to the increasing deployment of storage clusters across geo-distributed locations, leading to discrepancies in metadata synchronization across nodes.
Innovation Solution
A method for synchronizing metadata across federation member nodes by detecting discrepancies between slave and master metadata collections, computing a local metadata namespace root delta, and applying a set of commits to correct these discrepancies, implemented through a peer-to-peer metadata federation architecture with event-based synchronization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a centralized metadata management architecture is used to maintain consistency across nodes, then metadata synchronization reliability is improved, but the system creates central points of failure and reduces scalability
Solution Approach 1:
The patent segments the centralized metadata management into distributed metadata collections across multiple federation member nodes. Each node maintains a local metadata collection that can be independently managed and synchronized with peers, eliminating the single point of failure while maintaining consistency through distributed collaboration.
Solution Approach 2:
The patent transitions from a single-dimensional centralized hierarchy to a multi-dimensional peer-to-peer network structure. Metadata can be accessed and synchronized through multiple paths and nodes simultaneously, adding dimensional redundancy that improves reliability while distributing complexity across the network rather than concentrating it.
2Productivity
If storage clusters are deployed across geo-distributed locations to improve scalability, then system capacity and availability are enhanced, but metadata synchronization discrepancies increase between nodes
Solution Approach 1:
The patent implements feedback mechanisms where federation member nodes continuously exchange metadata collection information and detect discrepancies. When inconsistencies are identified between distributed nodes, the system triggers synchronization operations to restore consistency, enabling geo-distributed deployment while maintaining metadata reliability.
3Measurement precision
If frequent metadata synchronization is performed across all nodes to maintain consistency, then metadata accuracy is improved, but network bandwidth consumption and synchronization time increase
Solution Approach 1:
The patent applies partial synchronization by having nodes exchange only the metadata collection information rather than complete metadata sets. This selective synchronization approach maintains metadata accuracy while significantly reducing network bandwidth consumption and synchronization time compared to full synchronization.
Solution Approach 2:
The patent extracts and transmits only the essential metadata collection information (such as collection identifiers, version data, and change markers) between nodes rather than synchronizing entire metadata collections. This extraction approach enables frequent synchronization with minimal data transfer, maintaining accuracy while reducing time and bandwidth overhead.
Data Source
AI summary
A peer-to-peer global namespace for storage system metadata federations. Specifically, the disclosure proposes a peer-to-peer metadata federation architecture, where enterprise storage metadata for any and all geo-distributed sites may be distributed across and continually reconciled by a network of nodes, instead of being maintained by a single node assigned to a given geographic location. In implementing a federation, the network of nodes may lack exploitable, centralized points of vulnerability, and therefore, may exhibit no central point of failure. Furthermore, the disclosure proposes the implementation of a global metadata namespace for the enterprise storage metadata spanning the entirety of an enterprise (across all geographic locations). The global metadata namespace may be maintained on each node through event-based synchronization.