Asynchronous Database Replication for Committed Transaction Safety
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Asynchronous database replication can result in the loss of committed transactions during primary node failures, as secondary nodes may not have the latest data, leading to unrecoverable data loss and rollbacks.
Innovation Solution
Implementing a method where asynchronous replication logic is run as a standalone process across multiple nodes, allowing for the replication of committed transactions from a primary database node to secondary nodes, even after a primary node failure, ensuring data consistency and availability by electing a new primary node and replaying journal logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If asynchronous replication is used to improve performance, then write speed is improved, but data reliability deteriorates because committed transactions may be lost during primary node failure
Solution Approach 1:
The system performs preliminary actions by continuously replicating committed transactions to standby nodes before a failure occurs. The replication process operates in the background, ensuring that up-to-date copies are prepared in advance, so that when a primary node fails, the standby nodes already have the latest committed transactions and can immediately take over without data loss.
Solution Approach 2:
The invention creates a cushioning mechanism by maintaining multiple standby nodes with replicated data. This cushioning layer of redundant data protects against data loss during failures. The system buffers committed transactions across multiple nodes, providing a safety margin that ensures data availability even when the primary node fails unexpectedly.
2Reliability
If synchronous replication is used to ensure data consistency, then data reliability is improved, but write performance deteriorates due to conflict prevention requirements
Solution Approach 1:
The replication process is segmented into two independent phases: first, the primary node commits transactions locally without waiting for replication confirmation; second, the transactions are asynchronously replicated to standby nodes. This segmentation allows the commit operation to complete quickly without being blocked by replication delays, while still ensuring eventual consistency across the cluster.
Solution Approach 2:
The invention introduces an intermediary mechanism using journal logs and replication threads that mediate between the primary node and standby nodes. The journal log serves as an intermediary buffer that decouples the commit operation from the replication process, allowing the primary node to commit transactions immediately while the replication thread handles data transfer to standbys in the background.
3Device complexity
If replication logic is integrated into database nodes, then system complexity is reduced, but resource competition increases affecting replication performance
Solution Approach 1:
The replication logic is extracted from the main database node processes and implemented as separate, dedicated replication threads or processes. This extraction separates the replication function from the primary database operations, allowing replication to execute independently without competing for CPU, I/O, or memory resources with transaction processing, query execution, and other database activities.
Solution Approach 2:
The replication mechanism is designed to be self-service, with dedicated replication threads that autonomously manage their own resource allocation and execution. These threads independently handle journal log reading, data serialization, network transmission, and application to standby nodes without requiring resources from the main database server processes, effectively making replication a self-sufficient operation.
Data Source
AI summary
In an approach to improve asynchronous replication in database clusters embodiments retain committed transactions during database cluster replication. Embodiments build and run asynchronous replication logic using database nodes running as one or more standalone nodes, wherein the asynchronous replication logic is responsible for data replication from a primary database node to one or more secondary database nodes. Embodiments register a replication reader to a sender node to replicate journal logs and replay the journal logs on the one or more secondary database nodes. Further, embodiments monitor replication lag metrics from the sender node and notify a replication source and a replication target of a generated replication plan, wherein the cluster controller instructs the replication source and the replication target to be ready to initiate the replication plan. Additionally, embodiments adjust a replication plan to allow the asynchronous replication logic to catch up and complete uncommitted transactions.