Direct-to-Cloud File Migration for Large Active Enterprise Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Initial data migration in cloud-native global file systems is often slow and complex due to the larger data set being migrated exceeding the cache and copy-on-write management limitations, especially when the enterprise is using the data during migration.
Innovation Solution
Implement a direct-to-cloud migration approach using a downloadable virtual machine that transforms data into a cloud-compatible format and directly pushes it to the object store, allowing multiple passes to complete the migration efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the filer performs multiple snapshots for data migration, then the migration can be completed, but the migration speed decreases and complexity increases
Solution Approach 1:
A migration tool is introduced as an intermediary component between the source filesystem and the cloud object store. This tool performs the actual migration operations by reading from the source filesystem and writing to the cloud store, eliminating the need for the filer to perform complex snapshot operations. The migration tool manages the migration process independently, reducing the filer's processing burden and simplifying the overall migration complexity.
Solution Approach 2:
The migration process is segmented into distinct phases: a first pass that migrates the bulk of the data, and subsequent passes that handle changes detected between passes. This segmentation allows the migration to proceed in manageable stages, improving overall migration speed while reducing complexity by breaking down the large migration task into smaller, more manageable portions.
2Quantity of substance
If the data set is larger than the cache and COW, then more data can be migrated, but the migration becomes slower and more complex
Solution Approach 1:
The migration tool serves as an intermediary that handles data sets larger than the filer's cache capacity. Instead of relying on the filer's limited cache and copy-on-write mechanisms, the migration tool directly reads from the source filesystem and writes to the cloud object store, enabling migration of large data sets without being constrained by the filer's local caching capabilities.
Solution Approach 2:
The migration approach transitions from a local cache-based dimension to a cloud-based dimension. By using the cloud object store as the target destination, the system can handle data sets that exceed local storage capacity, effectively moving the storage constraint from the local filer to the cloud infrastructure.
3Productivity
If the enterprise uses data during migration, then productivity is maintained, but migration reliability decreases
Solution Approach 1:
The migration tool enables continuous migration operations without requiring the filer to stop processing data. The tool reads from the source filesystem and writes to the cloud object store concurrently with normal filer operations, maintaining business productivity while completing the migration. This continuous operation mode allows the filer to remain operational throughout the migration process.
Solution Approach 2:
The migration tool implements feedback mechanisms to track and report migration progress, allowing administrators to monitor the migration status in real-time. This feedback capability enables better management of the migration process during concurrent operations, ensuring that data usage during migration does not compromise reliability.
Data Source
AI summary
A cloud-native global file system to provide primary file storage for enterprise data is augmented to provide “direct-to-cloud” (D2C) migration of a data set. In addition to providing a filer, the service provider configures a direct-to-cloud migration tool executed in association with the hardware of the enterprise filesystem whose data set is being migrated. The migration tool reads the existing filesystem, transforms the data into a format consistent with that done by a filer, and pushes the result directly to an object store in the cloud. In a first pass, typically a large bulk of the data is moved and, depending on how the migration tool is configured, one or more additional passes may then be carried out. When migration is complete (or substantially complete), the filer then is activated to complete the migration (if need be) and take over the go-forward operations.