Semi-Sync Data Replication Using Metadata Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods, such as sync and async replication, face challenges in balancing data consistency and performance, with sync replication ensuring zero data loss but degrading performance and async replication potentially leading to inconsistent data and long Recovery Point Objective (RPO).
Innovation Solution
A semi-sync replication method using metadata logs and data logs to determine overlaps between target and previous input/output (IO) operations, allowing for efficient data replication by writing target IOs to source data volumes only after overlap replication, and utilizing metadata and data logs to ensure consistency and reduce log size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sync replication is used to ensure zero data loss, then data consistency is improved, but host IO performance deteriorates
Solution Approach 1:
The patent segments the IO operation into two independent parts: the host IO completion (writing to source data volume) and the replication IO (writing to target data volume). By using the metadata log to track and manage replication IOs separately, the host IO no longer needs to wait for replication completion, thus resolving the contradiction between data consistency and host IO performance.
Solution Approach 2:
The patent introduces a metadata log as an intermediary component that records IO operations and their status. This metadata log acts as a mediator between the host IO system and the replication system, allowing the host to proceed without waiting for replication while still ensuring data consistency through the metadata tracking mechanism.
2Productivity
If async replication is used to improve performance, then host IO performance is improved, but data consistency deteriorates
Solution Approach 1:
The patent implements a feedback mechanism through the metadata log that tracks the status of replication IOs. The system continuously monitors whether replication has completed and uses this feedback information to determine when data is consistent, thus maintaining data reliability while allowing async replication to improve host IO performance.
Solution Approach 2:
The patent performs preliminary actions by recording metadata about IO operations in the metadata log before actual replication occurs. This preliminary metadata recording allows the system to track and verify data consistency without blocking host IO operations, resolving the contradiction between performance and consistency.
3Reliability
If frequent snapshots are used to reduce RPO in async replication, then Recovery Point Objective is improved, but source data volume performance deteriorates
Solution Approach 1:
The patent uses a metadata log that records IO operations rather than creating frequent snapshots of the entire data volume. This copying approach at the metadata level allows the system to track data changes and achieve low RPO without the performance overhead of frequent full snapshots, thus resolving the contradiction between RPO and source data volume performance.
Solution Approach 2:
The patent extracts only the necessary metadata information (IO operations and their status) from the data volume changes, rather than copying entire snapshots. This extraction approach minimizes the impact on source data volume performance while still enabling accurate tracking for RPO calculation and data consistency verification.
4Reliability
If all target IOs are replicated to ensure data consistency, then data consistency is improved, but log size increases
Solution Approach 1:
The patent applies local quality by recording only the specific metadata information necessary for tracking replication status (such as IO operation identifiers, timestamps, and status flags) rather than duplicating entire data blocks. This selective metadata recording maintains data consistency verification capability while minimizing log size growth.
Solution Approach 2:
The patent implements a mechanism where metadata log entries are discarded (removed) after their purpose is fulfilled - specifically, after the associated replication IO completes or fails. This discarding of processed metadata entries prevents unbounded log growth while maintaining the ability to verify data consistency for active and recent operations.
Data Source
AI summary
Techniques for data replication involve determining, based on a metadata log, whether an overlap exists between a target input/output operation or target IO and previous IOs, wherein the metadata log records metadata related to data replication. Such techniques further involve writing the target IO to a source data volume according to a determination that no overlap exists between the target IO and the previous IOs. Such techniques further involve replicating data within the range of the overlap from the source data volume to a data log according to a determination that the overlap exists between the target IO and the previous IOs, and writing the target IO to the source data volume after completion of the replicating. Such techniques further involve replicating the target IO to a target data volume based on the metadata log and the data log.