Bootstrapping New Nodes in Distributed Memory Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed memory systems face inefficiencies in bootstrapping new nodes due to high resource usage and visibility issues, with conventional methods requiring complex synchronization and buffering, leading to potential data loss and system unavailability.
Innovation Solution
A method for bootstrapping a new node in a distributed memory network involves creating an empty key-value map and using a tombstone set to manage changes, allowing the node to start with the most recent data while concurrently receiving updates, reducing the need for buffering and minimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If time stamping is used for data synchronization between nodes, then data consistency can be maintained, but enormous synchronization problems occur
Solution Approach 1:
The patent extracts the timestamp mechanism from the synchronization process and replaces it with a push-based change notification system. Instead of relying on timestamps to determine data freshness, the master node directly pushes changes to replica nodes, eliminating the complexity of timestamp-based synchronization while maintaining data consistency.
Solution Approach 2:
The patent introduces a change notification mechanism as an intermediary between the master node and replica nodes. This mediator system tracks changes at the master node and automatically propagates them to replicas, replacing the need for complex timestamp comparisons and synchronization protocols.
2Reliability
If a stop the world approach is used to replicate key/value stores, then complete copies of key/value pairs can be provided to new nodes, but the entire distributed memory network is temporarily taken offline
Solution Approach 1:
The patent prepares replica nodes for data reception before the master node is taken offline. By pre-configuring the replica nodes with the necessary data structures and change notification handlers, the system can perform data replication during the stop-the-world period without extending downtime, and then quickly resume normal operations.
Solution Approach 2:
The patent enables continuous data replication by implementing a push-based change notification system that operates asynchronously. Once the initial data copy is provided to the new node, changes continue to be replicated in real-time without requiring the system to stop, maintaining continuous useful action during and after the bootstrapping process.
3Reliability
If complete copies of key/value pairs are provided to new nodes, then data consistency is achieved, but enormous resource usage and buffering requirements occur
Solution Approach 1:
The patent segments the data replication process into two phases: initial bulk data copy and subsequent incremental change propagation. This segmentation allows the system to provide complete data consistency while avoiding the need to buffer all changes simultaneously, as only incremental changes need to be buffered during the replication process.
Solution Approach 2:
The patent changes the parameter of data replication from synchronous bulk copying to asynchronous incremental propagation. By transforming the replication mechanism to push changes incrementally as they occur, the system achieves data consistency without requiring enormous buffering capacity to hold all changes until replication is complete.
Data Source
AI summary
A provisioned node, which is to be added into a distributed memory network, creates an empty provisioned key/value map. A master node, which is an existing node, in the distributed memory network, maintains existing key/value entries in an existing key/value map. An update cycle is performed, which includes receiving, by the master node and the provisioned node, one or more changes to the existing key/value entries. For any change that is a remove operation, the provisioned node stores a key specified in the remove operation in a tombstone set, and further removes the specified key from the provisioned key/value map. For any change that is a put operation of a specified key/value pair, the provisioned node puts the key/value pair in the provisioned key/value map.