Asynchronous Data Replication via Transaction Log Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication systems, such as those using Distributed Replicated Block Devices (DRBD), face issues with network unreliability and latency in wide area networks, leading to inconsistencies and service interruptions when network bandwidth drops below required levels.
Innovation Solution
A system that buffers write requests in volatile memory, records changes in a transaction log file, and replays log entries asynchronously, ensuring data consistency and integrity across storage devices, even in the presence of unreliable network links, by using a Lamport clock model for timestamping and allowing log entries to be replayed in total order.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous replication is used to ensure data consistency, then data integrity is improved, but network bandwidth requirements increase and system performance deteriorates in wide area networks
Solution Approach 1:
The system performs preliminary actions by buffering write requests in volatile memory and logging them to disk before transmission. This allows the system to prepare data for replication in advance, acknowledging writes locally without waiting for remote confirmation, thus maintaining data integrity while improving performance.
Solution Approach 2:
The patent introduces an intermediary logging mechanism that decouples the write acknowledgment from the actual data transmission. The log file acts as a mediator, allowing the primary storage to acknowledge writes immediately while the secondary storage processes them asynchronously, resolving the contradiction between reliability and performance.
2Reliability
If synchronous replication is used to maintain data consistency, then data availability is improved, but network latency increases causing application hangs or stalls
Solution Approach 1:
The system performs preliminary logging of write requests to disk before attempting network transmission. This preliminary action ensures that data is safely recorded locally, allowing the application to proceed without waiting for remote replication confirmation, thus reducing latency while maintaining availability.
Solution Approach 2:
The system dynamically adjusts the replication process by separating the acknowledgment phase (local logging) from the transmission phase (network send). This dynamic approach allows the system to remain responsive to applications while still performing replication in the background, reducing perceived latency.
3Reliability
If network bandwidth drops below required levels for synchronous replication, then data consistency is maintained, but service interruptions and application stalls occur
Solution Approach 1:
The system creates a copy of the write request in the log file and transmits this copy asynchronously to the secondary storage. This copying mechanism allows the primary storage to continue serving applications without waiting for the secondary storage to process the data, maintaining service continuity while ensuring consistency through the logged copy.
Solution Approach 2:
By performing the logging action preliminarily and independently of network conditions, the system ensures that data consistency is maintained through the log file while service continuity is preserved through immediate local acknowledgment. The network transmission becomes a separate, non-blocking operation.
4Reliability
If DRBD is used for distributed storage replication, then spatial redundant storage is provided, but the system requires high constant data throughput and managed network infrastructure
Solution Approach 1:
The system dynamically adapts to varying network conditions by using asynchronous transmission with acknowledgment-based flow control. Instead of requiring constant high throughput, the system adjusts its transmission rate based on network availability, making it versatile enough to operate in both managed and unmanaged network environments while maintaining spatial redundancy.
Solution Approach 2:
The patent changes the operational parameters of replication from synchronous (requiring high constant throughput) to asynchronous with selective acknowledgment. This parameter change allows the system to function in diverse network environments, including those with variable bandwidth and unmanaged infrastructure, while preserving data redundancy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Asynchronous data replication between a primary computing device and one or more secondary computing devices interconnected by a long distance network. The primary computing device buffers one or more write requests in volatile memory and appends the request to a local log file and performs data write back in background. This log file is transferred and replicated to the secondary devices. The so transferred and replicated log file is subsequently applied to block level storage means of the secondary computing devices.