Cross-Cluster Replication Using Operation History Checkpoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently replicating data across distributed clusters, particularly in scenarios where clusters are geographically separated or require disaster recovery, due to the impracticality of regular full data copies and the complexity of active-active replication models.
Innovation Solution
A method for cross cluster replication involving a leader cluster that maintains a history of operations with sequence numbers, allowing a follower cluster to request and apply these operations dynamically, with mechanisms for rolling back incomplete operations and using active-passive replication to maintain data consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If regular full data copies are performed for replication, then data consistency is maintained, but replication overhead and time consumption increase significantly
Solution Approach 1:
The patent extracts only the essential replication information (operation history and global checkpoint) from the leader cluster and transmits it to the follower cluster, rather than copying entire data sets. This selective extraction maintains data consistency while dramatically reducing replication time and overhead.
Solution Approach 2:
The replication process is segmented into discrete operations with sequence numbers, allowing the follower cluster to apply changes incrementally rather than requiring full data copies. Each operation is independently tracked and applied in order, ensuring consistency without time-consuming bulk transfers.
2Reliability
If active-active replication model is used, then high availability is improved, but system complexity increases
Solution Approach 1:
Instead of implementing complex active-active replication where both clusters simultaneously serve requests and coordinate changes, the patent inverts the model to active-passive: the leader cluster actively processes operations while the follower cluster passively replicates state. This inversion maintains high availability through failover capability while significantly reducing replication model complexity.
3Reliability
If follower cluster requests history frequently to catch up with leader cluster, then data consistency is maintained, but network overhead and processing load increase
Solution Approach 1:
The follower cluster uses feedback based on the global checkpoint to intelligently determine when to request history. By comparing its current checkpoint against the leader's checkpoint, the follower only requests updates when actually behind, avoiding unnecessary network traffic while maintaining consistency. This feedback-driven approach reduces network overhead compared to frequent periodic requests.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is provided a computer-implemented method for updating history and a global checkpoint. The computer-implemented method comprises receiving (510) an operation by a leader cluster (410A), the operation changing data in a primary shard of the leader cluster; assigning (520) a sequence number to the operation; incrementing the sequence number; executing (530) the operation; evaluating (540) completion of the operation; when the operation is not successfully completed, rolling back (550) the operation; when the operation is successfully completed, storing (560) the operation and the sequence number in the history of the leader cluster (410A); and updating (570) the global checkpoint to the sequence number associated with the successfully completed operation. There is further provided a corresponding system.