Storage Cluster Migration Using Dual Writes and Snapshot Sync
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data migration methods in cloud computing environments, such as those using Elastic Block Storage (EBS), suffer from long IO request interruption times and increased resource consumption due to the need for data synchronization, especially in high-performance scenarios, leading to noticeable delays and architectural complexity.
Innovation Solution
Implementing a dual write mode for block storage access clients, combined with incremental data synchronization and snapshot data generation, to migrate data between storage clusters while minimizing IO request interruptions by pausing them at the microsecond level.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data migration is performed by continuously comparing data between block devices and performing data synchronization, then data consistency is ensured, but IO request interruption time increases and user-perceived delay becomes long
Solution Approach 1:
The system performs preliminary data synchronization before the final migration step. The block device is first migrated to a temporary location with data synchronized, then finally migrated to the target storage cluster. This preliminary action allows the system to prepare data in advance, reducing the interruption time during the actual migration by only requiring a brief pause for the final switch.
Solution Approach 2:
The data migration process is divided into multiple stages: initial data synchronization to a temporary location, incremental data updates during the transition period, and final migration completion. This segmentation allows the migration to proceed without requiring long interruptions, as each stage can be completed independently with minimal user-perceived delay.
2Reliability
If traditional data migration methods are used with continuous data comparison and synchronization, then data consistency is maintained, but resource consumption increases and architectural complexity increases
Solution Approach 1:
A temporary block device location is introduced as an intermediary in the migration process. Data is first synchronized to this temporary location, then finally migrated to the target storage cluster. This intermediary approach simplifies the migration architecture by providing a clear transition path and reducing the complexity of direct source-to-target synchronization while maintaining data consistency.
3Reliability
If data migration is performed with continuous synchronization, then data consistency is ensured, but system performance decreases due to resource consumption
Solution Approach 1:
Instead of continuous data comparison and synchronization, the system uses periodic incremental data updates. After the initial migration to the temporary location, only changed data blocks are updated in the background during normal operation. This periodic action significantly reduces resource consumption and maintains system performance while ensuring data consistency is achieved.
Data Source
AI summary
The present disclosure discloses a data migration method and a system based on a storage cluster, and a storage medium. The method includes: triggering a dual write mode of a block storage access client in response to a data migration instruction, and implementing incremental data synchronization to a source storage cluster and a target storage cluster by executing the dual write mode, where the data migration instruction is used to instruct to migrate data in the source storage cluster to the target storage cluster; and triggering a storage node in the source storage cluster to generate corresponding snapshot data for existing data in response to the data migration instruction, and migrate the snapshot data to the target storage cluster.