Distributed Metadata Synchronization via Delegate Node Snapshot Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In partition-tolerant distributed key/value systems, the reconciliation and synchronization of metadata when a node rejoins a cluster can create a bottleneck for the system, as the master node handles all synchronization tasks, leading to potential performance issues.
Innovation Solution
The master node selects a delegate node using a load balancing algorithm to transmit a snapshot of its key/value store to the rejoining node, distributing the synchronization load and enabling efficient merging of metadata, thereby reducing the burden on the master node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the master node handles all synchronization tasks when a node rejoins the cluster, then the key/value system maintains consistency across the cluster, but the master node becomes a performance bottleneck
Solution Approach 1:
The patent divides the synchronization task into two phases: a fast snapshot transfer phase handled by a delegate node, and a metadata reconciliation phase handled by the master node. This segmentation allows the bulk data transfer to be distributed while the master node focuses on maintaining consistency, resolving the contradiction between reliability and productivity.
Solution Approach 2:
The patent introduces a delegate node as an intermediary to handle the snapshot transmission to rejoining nodes. This intermediary absorbs the heavy lifting of data transfer, preventing the master node from becoming a bottleneck while still ensuring eventual consistency through the master-coordinated reconciliation phase.
2Reliability
If the master node transmits the entire key/value store to a rejoining node, then the node is fully synchronized, but the transmission time and network bandwidth consumption increase significantly
Solution Approach 1:
The patent performs preliminary action by creating and storing snapshots of the key/value store at delegate nodes before rejoining occurs. When a node rejoins, it can quickly receive a pre-prepared snapshot rather than waiting for the master node to generate and transmit one, significantly reducing rejoining time while maintaining synchronization completeness.
Solution Approach 2:
The patent uses snapshot copying as the primary synchronization mechanism. Instead of transmitting the entire key/value store from the master node, a copy (snapshot) is created at a delegate node and transmitted to the rejoining node. This copying approach reduces transmission time and master node involvement while ensuring the rejoining node receives an accurate copy of the cluster state.
3Reliability
If the master node reconciles all metadata updates from rejoining nodes, then the cluster maintains a single source of truth, but the master node's processing load increases
Solution Approach 1:
The patent segments the metadata reconciliation process into snapshot-based synchronization (handled by delegate nodes) and incremental metadata updates (handled by the master node). This segmentation reduces the master node's processing load by offloading the bulk data synchronization to delegate nodes while maintaining consistency through coordinated metadata reconciliation.
Solution Approach 2:
The delegate node acts as an intermediary in the reconciliation process, handling the initial snapshot synchronization and reducing the volume of data that needs to be reconciled by the master node. The master node then focuses on coordinating the reconciliation of metadata updates, reducing its processing load while maintaining the single source of truth.
Data Source
AI summary
Exemplary methods, apparatuses, and systems include a master node receiving, from a new node, a request indicating the new node is joining a cluster of nodes. The master node transmits, to the new node, an identifier of a delegate node within the cluster. The master node receives, from the delegate node, a request to maintain a synchronization point while the delegate node creates and transmits a snapshot of cluster metadata. The master node receives, from the new node, an indication that the new node is ready to join the cluster. Upon determining that the cluster has synchronized metadata following the synchronization point of the delegate node and prior to the ready indication, the master node transmits an acknowledgment to the new node to cause the new node to transition to an active state within the cluster.