Geo-replicated Database Reconciliation via Distributed State Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-availability database services face challenges in reconciling databases across geo-replicated clusters, especially in situations like unavailability of primary databases, communication link failures, and concurrent operations, which can prevent synchronization and lead to data inconsistencies.
Innovation Solution
A stateless messaging protocol managed by a distributed cluster management module uses reconciliation data structures to track database states and operations, allowing each cluster to determine whether changes can be reconciled, ensuring that databases remain synchronized without relying on a central control point.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a coordination infrastructure is used to ensure data consistency between geo-replicated clusters, then data consistency is improved, but the system becomes more complex and introduces a single point of failure
Solution Approach 1:
The coordination infrastructure is segmented and distributed across multiple nodes rather than centralized. Each node independently maintains reconciliation data structures and processes reconciliation requests, eliminating the single point of failure while preserving data consistency through distributed coordination.
Solution Approach 2:
Reconciliation data structures serve as intermediaries between clusters, carrying state information and operation metadata. These data structures enable indirect coordination between clusters without requiring a centralized coordinator, reducing system complexity and improving reliability.
2Reliability
If databases are geo-replicated across different locations for disaster recovery, then availability is improved, but maintaining synchronization between clusters becomes more difficult
Solution Approach 1:
Reconciliation data structures are pre-established and maintained at each cluster before conflicts occur. These data structures continuously track database state and operation metadata, enabling rapid conflict detection and resolution without complex real-time synchronization protocols when failures or concurrent operations occur.
3Productivity
If concurrent operations are allowed on primary and secondary databases independently, then productivity is improved, but data consistency becomes harder to maintain
Solution Approach 1:
The system implements feedback through reconciliation data structures that continuously monitor and compare database states across clusters. When concurrent operations create conflicts, the feedback mechanism detects state divergences and triggers reconciliation processes, allowing high concurrency while maintaining eventual consistency through automated conflict resolution.
Data Source
AI summary
A database associated with a primary database cluster may be replicated in a backup database cluster located in a different location in order to provide a highly-available fault tolerant database service. The databases are reconciled through a cluster management module distributed in each database cluster. The cluster management module uses a set of reconciliation data structures to track locally the reconciled states of each database in each database cluster, the operations made locally to the databases in each database cluster, and the author of the operations. The cluster management module in each database cluster engages in a stateless messaging protocol using the set of reconciliation data structures to determine whether or not the databases may be reconciled.