Storage Cluster Data Migration via Exception Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face inefficiencies and performance degradation during the process of adding new nodes for horizontal capacity expansion, as existing data migration methods are lengthy, cumbersome, and disruptive, requiring significant downtime and resource utilization.
Innovation Solution
A system and method for efficiently migrating data between storage nodes using an exception table and background process, allowing for simultaneous I/O operations and node functionality during migration, with exception tables identifying areas to be migrated and redirecting I/O requests to ensure data integrity and minimize downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If data is re-striped across all nodes when a new node is added, then data is properly distributed across the expanded cluster, but the migration process takes several days and drastically decreases cluster performance
Solution Approach 1:
The patent divides the data migration process into segments by creating exception tables that identify only the specific zones requiring migration. Instead of re-striping all data across the cluster, the system segments the migration task to only affect zones that need to be moved to the new node, dramatically reducing migration time while maintaining proper data distribution.
Solution Approach 2:
The system performs preliminary actions by creating exception tables that pre-identify which zones need migration before the actual data movement begins. This preliminary identification allows the system to prepare migration paths and minimize disruption to ongoing I/O operations, reducing the overall migration time and performance impact.
2Manufacturing precision
If data migration is performed using current methodologies, then data is redistributed across the expanded cluster, but cluster performance is drastically decreased during the process
Solution Approach 1:
The patent applies partial action by migrating only the necessary portions of data (zones identified in exception tables) rather than performing a complete re-striping of all data. This selective migration approach maintains data distribution accuracy while minimizing the impact on cluster performance during the expansion process.
Solution Approach 2:
The exception table acts as an intermediary structure that guides the migration process. It identifies which zones need to be moved and to where, allowing the system to perform precise data redistribution without unnecessarily impacting performance. The exception table mediates between the need for accurate data distribution and the need to maintain cluster productivity.
3Reliability
If a complicated locking method is used to prevent data corruption during migration, then data integrity is maintained, but the migration process becomes more complex and time-consuming
Solution Approach 1:
The patent extracts the locking complexity from the overall migration process by using exception tables to pre-identify migration zones. This allows the system to apply locking mechanisms only where necessary (during actual data movement of identified zones) rather than implementing complex locking across the entire cluster, simplifying the overall process while maintaining data integrity.
4Manufacturing precision
If the entire data body is read and written during migration, then data is properly redistributed, but the process is lengthy and ineffective for rapid expansion
Solution Approach 1:
The patent extracts only the necessary data portions for migration by using exception tables to identify specific zones that need to be moved. Instead of reading and writing the entire data body, the system selectively migrates only the zones identified in the exception tables, achieving complete and accurate data redistribution while dramatically reducing migration time and improving expansion speed.
Data Source
AI summary
A method, system, apparatus, and computer-readable medium are described for the background movement of data between nodes in a storage cluster. According to one method, exception tables are generated that include data identifying the areas on each node that need to be migrated to another node and the areas on each node that are to be migrated from another node. The exception tables may be generated in response to the failure of a node in the storage cluster or in other circumstances. A background resynchronization thread utilizes the contents of the exception tables to migrate data between the nodes of the cluster, thereby synchronizing the data stored on the nodes. Input/output operations directed to an area that is to be migrated from another node are redirected to the other node. The methods, systems, apparatus, and computer-readable medium may be utilized with operations including the rebuilding, expansion, contraction, and compaction of a storage cluster.