Data Migration System for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, data migration processes often face challenges in maintaining data persistency, minimizing performance impact, and ensuring high availability, while also allowing for quick roll-back in case of interruptions, especially during operations like load balancing, storage expansion, or replication.
Innovation Solution
A computerized method and system for data migration that involves scanning data addresses, applying a new ownership function, and writing data to new storage areas while maintaining copies according to both old and new ownership functions, allowing asynchronous execution of I/O requests and enabling quick roll-back by keeping original data intact until migration is complete.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is migrated from first storage area to second storage area using new ownership function, then data distribution and load balancing are improved, but system complexity and risk of data inconsistency increase
Solution Approach 1:
The system performs preliminary scanning of data addresses using the new ownership function before actual migration begins. This preliminary action identifies all data that needs to be migrated and prepares the migration plan, allowing the system to execute the migration systematically while maintaining data consistency through predetermined migration paths and timing.
Solution Approach 2:
The migration process is divided into distinct segments: scanning phase, identification phase, and execution phase. During the scanning phase, data addresses are scanned and categorized. In the identification phase, specific data destinations are determined using the new ownership function. The execution phase handles actual data transfer with proper timing controls. This segmentation reduces overall system complexity by breaking down the complex migration task into manageable, controlled steps.
2Reliability
If data migration is performed during normal I/O operations, then system availability is maintained, but data persistency and consistency may be compromised
Solution Approach 1:
The system implements periodic action by scanning data addresses at specific intervals and executing migrations during designated I/O intermission periods. The scanning occurs periodically to identify data needing migration, while actual migration executions are timed to occur during natural I/O pauses. This periodic approach maintains system availability by continuing normal operations while ensuring data consistency through controlled, timed migration events.
Solution Approach 2:
Before executing actual data migration, the system performs preliminary scanning and identification of data that needs to be migrated. This preliminary action ensures that all data addresses are known and categorized before migration begins, allowing the system to maintain data consistency by having a complete picture of migration requirements while continuing normal I/O operations.
3Quantity of substance
If data is migrated immediately without keeping original copies, then storage space is optimized, but risk of data loss increases during migration interruptions
Solution Approach 1:
The system implements beforehand cushioning by maintaining original data copies in the first storage area until migration is fully complete and verified. This cushioning period acts as a safety buffer, allowing the system to recover from migration interruptions by reverting to original copies if needed. Only after successful completion of migration does the system remove original copies, thus optimizing storage space while maintaining data safety throughout the process.
4Speed
If scanning and migration are performed continuously without I/O intermission, then migration speed is maximized, but performance impact on storage system increases
Solution Approach 1:
The system uses periodic action by performing scanning operations continuously in the background while executing actual data migrations during designated I/O intermission periods. This periodic execution pattern allows the system to maintain high migration speed over time while minimizing immediate performance impact, as migrations are timed to occur during natural pauses in I/O operations rather than continuously competing for resources.
Data Source
AI summary
The presently disclosed subject matter includes a data storage system and method for the purpose of migrating data in a distributed storage system comprising a plurality of separate storage areas.