Multi-threaded Shadow Migration for Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data migration in large-scale storage systems using multiple threads often results in saturated performance gains due to serialization of migration operations, leading to inefficient migration times, as simply adding more threads does not equate to increased performance.
Innovation Solution
The allocation of an optimal number of threads for data migration based on the degree of parallelism available, calculated using the derivative of the expected performance gain, ensures efficient migration by balancing serialized and parallelized operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple threads are employed for data migration, then migration speed is improved, but performance gain becomes saturated due to serialization of migration operations
Solution Approach 1:
The patent dynamically determines the optimal number of threads based on the degree of parallelism available for each migration operation. Instead of using a fixed number of threads, the system calculates the derivative of expected performance gain to identify the point where additional threads no longer provide proportional performance improvement, thereby avoiding saturation and optimizing migration time
Solution Approach 2:
The system changes the parameter of thread count from a static value to a dynamically optimized value. By calculating the derivative of expected performance gain with respect to the number of threads, the system identifies the optimal thread count that maximizes migration speed while avoiding the saturation point where serialization effects diminish returns
2Reliability
If the source file system is taken offline for migration, then data integrity is ensured, but system availability is reduced
Solution Approach 1:
The patent performs preliminary actions by creating shadow copies of the file system structure and metadata before actual data migration begins. This allows the source file system to remain online and operational during migration, as the shadow structure can be populated in advance and used to guide the migration process without requiring source unavailability
Solution Approach 2:
The system creates a shadow copy of the file system hierarchy including directories and metadata structures. This shadow structure is copied and maintained during the migration process, allowing the source to remain accessible while the target is being populated. The shadow structure serves as a template that guides the migration without requiring the source to be taken offline
Data Source
AI summary
Implementations described and claimed herein provide systems and methods for migration data from a source to a target in the background using an optimal number of threads. In one implementation, a file system operation request is received at the target. The file system operation request specifying a read request for a directory having a directory level migration attribute on the target that is marked. An optimal number of threads is allocated for migrating the directory. Metadata for content in the directory is obtained from the source using the optimal number of threads. A directory entry for a file in the directory is created on the target using the metadata, the directory entry for the file associated with a file level migration attribute that is marked.