Cross-Region Block Volume Replication with Snapshot Delta Checkpoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud-based data replication methods, such as periodic backups, suffer from significant latency, elevated computational demands, and increased resource expenses due to metadata overhead, leading to prolonged recovery point objectives (RPO) and recovery time objectives (RTO) during system failures.
Innovation Solution
Implementing asynchronous cross-region block volume replication by generating snapshots and deltas, allowing incremental data transfer and reducing the need for complete backups, thereby minimizing system interruptions and improving data redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If periodic backups are used for data replication, then data redundancy is achieved, but significant latency and prolonged RPO/RTO occur during system failures
Solution Approach 1:
The system creates snapshots at predetermined logical times and proactively generates deltas between snapshots before failures occur. This preliminary preparation of incremental data transfers enables rapid recovery by having recovery data ready in advance, significantly reducing RPO and RTO compared to traditional periodic backups that must execute when failures occur.
Solution Approach 2:
The replication system dynamically adjusts by generating incremental deltas between snapshots rather than performing static periodic full backups. This dynamic incremental approach allows the system to efficiently transfer only changed data blocks, reducing transfer time and improving recovery speed while maintaining data redundancy.
2Reliability
If complete backups are performed regularly, then data recovery capability is improved, but computational demands and resource expenses increase due to metadata overhead
Solution Approach 1:
The system extracts and transfers only the incremental deltas (changed data blocks) between snapshots rather than performing complete backups. This extraction approach eliminates the need to process and transfer unchanged data, significantly reducing computational overhead and resource expenses while maintaining full recovery capability through the combination of snapshots and incremental deltas.
Solution Approach 2:
Instead of performing excessive complete backups, the system applies partial action by transferring only the necessary incremental changes (deltas) between snapshots. This partial transfer approach reduces computational demands and metadata processing while achieving the same recovery capability with minimal resource consumption.
3Stability of the object's composition
If synchronous replication is used, then data consistency is maintained, but input-output operations and performance metrics are significantly impacted
Solution Approach 1:
The system uses periodic snapshots at logical times rather than continuous synchronous replication. This periodic approach allows normal input-output operations to proceed without the continuous overhead of synchronous replication, while still maintaining data consistency through the structured snapshot and delta generation process at predetermined intervals.
Solution Approach 2:
The replication system dynamically operates in the background without blocking normal input-output operations. By generating deltas asynchronously between snapshots and allowing I/O operations to continue during snapshot creation and delta generation, the system maintains data consistency while preserving productivity and performance metrics.
Data Source
AI summary
Techniques are provided for cross-region replication of block volume data. The techniques include a method wherein a computer system implements operations including creating a first snapshot of a block volume at a first geographic region and at a first logical time, the block volume including a plurality of partitions, transmitting first snapshot data to an object storage system at a second geographic region, creating a second snapshot of the block volume at the first geographic region and at a second logical time, generating a plurality of deltas, transmitting a plurality of delta data sets corresponding to the plurality of deltas to the object storage system at the second geographic region, generating a checkpoint at least in part by aggregating object metadata associated with the plurality of deltas and the first snapshot, receiving a restore request to generate a restore volume, and generating the restore volume from the checkpoint.