Asynchronous Shard Replication with Lead-Sync Failover Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database replication solutions fail to meet the requirements of high performance, low overhead replication with strong consistency, fast failover, full SQL support, and horizontal scalability, particularly in sharded databases.
Innovation Solution
Implement asynchronous database transaction replication using a Raft consensus protocol, where the leader server performs DML operations and commit operations asynchronously, while ensuring synchronous replication of commit log records, and utilizes replication units to optimize efficiency and ensure zero data loss during failover.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous database replication is implemented to ensure strong consistency, then data consistency is improved, but user transaction response time increases
Solution Approach 1:
The patent segments the replication process into two distinct phases: async log replication (fast, non-blocking) and commit synchronization (ensures consistency). This segmentation allows the leader to return transaction results immediately after logging without waiting for follower commits, while still ensuring eventual consistency through the commit phase.
Solution Approach 2:
The patent performs preliminary actions by replicating the transaction log to followers asynchronously before the commit is finalized. This preliminary log replication allows the system to prepare for commitment without blocking the user transaction, improving response time while maintaining consistency through subsequent commit verification.
2Loss of time
If asynchronous replication is used to reduce transaction response time, then user transaction response time is improved, but data consistency and reliability deteriorate
Solution Approach 1:
The patent implements feedback mechanisms where followers send commit acknowledgments back to the leader. The leader tracks commit status and can detect inconsistencies, triggering recovery procedures. This feedback loop ensures that asynchronous replication maintains strong consistency by verifying commit status across the cluster.
Solution Approach 2:
The patent prepares for potential consistency issues by implementing recovery mechanisms in advance. When the leader fails, followers that have committed transactions can become new leaders and continue the replication, ensuring no data loss. This beforehand cushioning allows asynchronous operation while guaranteeing consistency through pre-prepared recovery paths.
3Reliability
If synchronous commit replication is implemented to ensure zero data loss, then data loss prevention is improved, but replication performance and throughput deteriorate
Solution Approach 1:
The patent implements dynamic replication behavior where the commitment process adapts based on system state. During normal operation, commits are synchronized to ensure zero data loss. During leader failure scenarios, the system dynamically allows followers with committed logs to become new leaders, maintaining zero data loss while preserving throughput through flexible leadership transitions.
4Reliability
If the leader waits for follower commit acknowledgments before returning transaction results, then data consistency is improved, but user transaction response time increases
Solution Approach 1:
The patent performs preliminary log replication to followers before the commit phase, but returns transaction results to users immediately after the log is replicated without waiting for commit acknowledgments. This preliminary action separates the user-facing response from the consistency-guaranteeing commit process, achieving both fast response and strong consistency.
Data Source
AI summary
A lead-sync log record is used to synchronize the replication logs of follower shards to the leader shard. In response to a failure to determine that there is a consensus for a database transaction commit operation after a shard server becomes a new leader, the new leader shard performs a sync operation using the lead-sync log record to synchronize replication logs of the follower shards to the replication log of the new leader. A shard server identifies a first transaction having a first log record but not a post-commit log record in the replication log, defines a recovery window in the replication log starting at the first log record of the identified first transaction and ending at the lead-sync log record, identifies a set of transactions to be recovered, and performs a recovery action on the set of transactions to be recovered.