Data Transfer Coordinator for Continuous Replication Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer technologies between data centers often restrict user access to data during replication, which is problematic for time-critical applications, as users cannot access data while it is being transferred, leading to potential disruptions in services like medical or financial information.
Innovation Solution
Implementing a phased commitment approach for data transfer within and between data sites, where new data entries are directed to a new site, and each node iterates through data entries to send key-value pairs to a local site master, which then sends them to the new site, allowing users to access data continuously during replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is transferred between data centers using traditional replication methods, then data redundancy and security are improved, but user access to data is restricted during the replication process
Solution Approach 1:
The system segments data into key-value pairs and distributes them across multiple nodes within data centers. During replication, the data transfer coordinator divides the replication task into manageable chunks, allowing different nodes to handle different segments of data simultaneously. This segmentation enables continuous user access to other data segments while replication occurs.
Solution Approach 2:
A data transfer coordinator acts as an intermediary between source and destination data centers during replication. This coordinator manages the replication process, routing data through intermediate nodes and maintaining active connections, thereby allowing user access to continue uninterrupted while data is being replicated in the background.
2Reliability
If data replication is performed between data centers, then data security against catastrophic failures is improved, but service disruption occurs during the transfer process
Solution Approach 1:
The system establishes replication connections and prepares data transfer pathways before actual data replication begins. Nodes are pre-configured and ready to receive data, allowing the replication process to start immediately without service disruption. Users can continue accessing data while the replication infrastructure is already in place and actively transferring data.
Solution Approach 2:
The replication process is designed to operate continuously in the background without interrupting user access to data. The system maintains continuous data flow between source and destination centers while users simultaneously access data through the same system, ensuring that the useful action of data access continues uninterrupted throughout the replication process.
3Reliability
If traditional data transfer methods are used between data centers, then data redundancy is achieved, but access rates to data are reduced during transfer
Solution Approach 1:
The system adds a temporal dimension to data replication by operating in parallel layers: user access operations continue in the foreground while replication occurs in the background. This dimensional separation allows data access requests to be fulfilled from existing data centers while replication simultaneously populates new data centers, maintaining high access rates throughout the process.
Solution Approach 2:
The system dynamically adjusts data routing during replication, automatically directing user access requests to the nearest available data center while replication traffic flows through dedicated channels. This dynamic routing ensures that user access speed is maintained by always directing traffic to operational data centers, regardless of ongoing replication activities.
Data Source
AI summary
The method including, receiving a transfer command at a transfer coordinator node of a first node cluster at a first site to initiate a transfer of data from the first node cluster to a second node cluster at a second site. The method further including causing a status of the second site to be switched from an inactive status to an active status at the nodes of the first node cluster. The method further including initiating a transfer of the data from the first node cluster to the second node cluster. The method further including determining that the second node cluster has the active status at the nodes of the first node cluster and causing the data transaction request to be transferred from the nodes at the first node cluster to the second node cluster during the transfer of the data.