Live Filesystem Migration for Cloud Data Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual Big Data migration to the cloud is prone to data inconsistency, business disruption, and high IT resource requirements, making it challenging to ensure data accuracy and meet service level agreements (SLAs) during long-running migrations, especially with dynamically changing datasets.
Innovation Solution
An automated computer-implemented method and system for live migration that allows continuous operation during data transfer, ensuring data consistency across on-premises and cloud storage systems without manual scripting, scheduling, or transfer devices, using a depth-first scan approach to manage metadata migration and selectively apply changes to maintain data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual migration scripts are used to transfer big data to the cloud, then data can be migrated, but data inconsistency occurs during the lengthy migration process
Solution Approach 1:
The patent implements a pre-migration validation phase that performs integrity checks and consistency validation before the actual data transfer begins. This preliminary action ensures that the source data is in a consistent state and identifies potential issues before they affect the migration, thereby maintaining data consistency throughout the lengthy migration process without requiring manual validation during each stage.
Solution Approach 2:
The system implements continuous feedback mechanisms during migration that monitor data consistency in real-time. Validation rules are applied continuously, and any inconsistencies detected during the migration process trigger automatic corrective actions or alerts, ensuring data reliability is maintained throughout the extended migration duration without manual intervention.
2Reliability
If incremental migration strategies are used to transfer data over many months, then data can be migrated, but business disruption occurs due to handling updates
Solution Approach 1:
The patent implements dynamic migration scheduling that automatically adjusts migration operations based on real-time system conditions and business requirements. The system can pause, resume, and prioritize different data sets dynamically, allowing critical business operations to continue uninterrupted while migration proceeds in the background. This dynamic approach ensures SLA compliance without causing operational disruption.
Solution Approach 2:
The system introduces an intermediary layer that manages the interaction between migrating data and active business operations. This intermediary handles update conflicts, coordinates between source and target systems, and ensures that business-critical operations can proceed without interruption while data migration occurs in the background, maintaining both reliability and operational continuity.
3Extent of automation
If custom migration scripts are created and maintained manually, then data migration can be performed, but high IT resource requirements and increased costs occur
Solution Approach 1:
The patent implements a universal migration platform that handles multiple data types, formats, and target destinations through a single standardized interface. This multi-functional system eliminates the need for custom scripts for different scenarios, as the platform automatically adapts to various migration requirements through configuration rather than custom code, thereby reducing IT resource requirements while maintaining high automation levels.
Solution Approach 2:
The system implements self-service capabilities that automatically generate, optimize, and maintain migration configurations without requiring manual script creation or maintenance. The platform self-adjusts to changing requirements, automatically updates migration parameters, and manages its own operations, eliminating the need for dedicated IT resources to create and maintain custom migration scripts while reducing operational complexity.
4Measurement precision
If manual validation of migration accuracy is performed, then data consistency can be checked, but it is impracticable for large scale data sets
Solution Approach 1:
The patent implements segmentation of the validation process that divides large data sets into manageable segments for systematic verification. The system applies validation rules to each segment independently and aggregates the results, enabling accurate validation of petabyte-scale data sets by breaking them into smaller, processable units that can be validated efficiently without overwhelming computational resources.
Solution Approach 2:
The system implements a multi-layered validation approach that applies different levels of checking based on data criticality. Critical data receives exhaustive validation, while less critical data receives sampling-based validation. This partial action strategy ensures high measurement precision for important data while making the overall validation process scalable to large data volumes by not applying the same level of scrutiny uniformly across all data.
Data Source
Figure 1A~1C
Figure 2
Figure 3
AI summary
A computer-implemented method of migrating metadata from a donor filesystem D having a rooted tree structure to a beneficiary filesystem B while processing commands that operate on the metadata may comprise, while a command to operate on the metadata is not received, replicating the donor filesystem D at the beneficiary filesystem B by sequentially copying metadata of nodes of the donor filesystem D to the beneficiary filesystem B. When none of arguments are present in the beneficiary filesystem B, the command may be executed at the donor filesystem D only. When only some of the arguments are present in the beneficiary filesystem B, the command may be enqueued at least until all arguments of the command are present in the beneficiary filesystem B. When all arguments thereof are present in the beneficiary filesystem B, the enqueued commands may be dequeued and scheduled for execution.