Asynchronous Data Replication Bucketing Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing asynchronous replication systems face challenges in maintaining write order fidelity while retaining performance and bandwidth benefits, often resulting in data loss and inconsistent recovery during fail-over scenarios due to buffering and compression techniques.
Innovation Solution
The implementation of a bucketing mechanism where write operations are organized into buckets at the primary server, allowing for asynchronous replication to a secondary server while ensuring write order fidelity, utilizing a write-back cache and zero buffer copy mechanism to eliminate duplicate writes and maintain data consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If asynchronous replication uses buffering and compression techniques, then bandwidth efficiency and performance are improved, but write order fidelity deteriorates resulting in data loss and inconsistent recovery
Solution Approach 1:
The patent segments the data stream into discrete write operations, each assigned a unique sequence number. This segmentation allows the system to maintain individual write order tracking while still enabling batched asynchronous replication. Each write operation is treated as an independent unit that can be buffered and compressed without losing its positional context in the original data stream.
Solution Approach 2:
The patent introduces an intermediary mapping table that tracks the correspondence between source and destination write operations. This intermediary structure acts as a mediator between the asynchronous replication process and the original write order requirements, allowing the system to reorder or verify writes at the destination while maintaining bandwidth efficiency through buffered asynchronous transmission.
2Productivity
If asynchronous replication buffers write operations, then performance is improved, but data consistency during fail-over deteriorates
Solution Approach 1:
The patent implements a feedback mechanism through acknowledgment signals sent from the destination back to the source. When a write operation is successfully applied at the destination, an acknowledgment is sent back, allowing the source to track which writes have been replicated. This feedback loop ensures that during fail-over, only committed and acknowledged writes are considered consistent, while buffered but unacknowledged writes are properly handled.
Solution Approach 2:
The patent performs preliminary actions by pre-allocating sequence numbers and creating mapping entries before actual data transmission. This preliminary setup allows the system to maintain write order fidelity without requiring complex real-time coordination during fail-over events, as the ordering information is already established in advance.
3Reliability
If synchronous replication is used to maintain write order fidelity, then data consistency is improved, but performance and bandwidth efficiency deteriorate
Solution Approach 1:
The patent employs periodic action by batching multiple write operations together for asynchronous replication instead of processing each write individually in real-time. This periodic batched approach allows the system to accumulate writes during normal operation (improving performance) and then replicate them as a group, maintaining overall write order fidelity while reducing the performance impact compared to strict synchronous replication.
Data Source
AI summary
Technologies are provided herein for asynchronous data replication. A primary server maintains and exposes a storage volume for use by storage clients. The primary server receives write operations directed toward the storage volume and performs the writes on the storage volume. The primary server also creates buckets containing each of the write operations. The buckets are opened at a first point in time and closed at a second point in time that may be selected based upon the contents of the buckets, a predefined time period, or based upon an indication from an application program. The buckets are transmitted to a second server computer asynchronously. When each bucket is received completely by the second server, its contents are written to a storage volume maintained by the second computer. The buckets may be created from pages of a write-back cache utilizing a zero buffer copy mechanism.