Granular Data Migration Using B+ Tree Snapshots and Tag Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in efficiently migrating files with high granularity while maintaining performance and consistency, as existing methods either lack fine-grained control or suffer from disruptive bulk filesystem transfers.
Innovation Solution
The implementation of a granular data migration process using B+ Trees and MTree structures allows for sub-directory level migration, employing tagging and snapshot differencing to efficiently move and replicate data, ensuring high granularity and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If per-file data movement is used, then fine-grained control is achieved, but throughput is slow
Solution Approach 1:
The patent segments the filesystem into smaller units called MTrees, which represent manageable portions of the filesystem. Each MTree can be independently migrated, allowing fine-grained control similar to per-file operations but with better throughput by operating on directory-level units rather than individual files. This segmentation enables parallel migration of multiple MTrees simultaneously.
2Productivity
If bulk filesystem transfer is used, then throughput is increased, but disruption occurs and fine-grained control is lost
Solution Approach 1:
The patent divides the filesystem into multiple MTrees, each representing a manageable segment. This allows bulk transfer of entire MTrees (achieving high throughput) while maintaining the ability to selectively migrate only specific MTrees that need to be moved (maintaining fine-grained control). The segmentation enables parallel operation of multiple data protection processes on different MTrees.
Solution Approach 2:
The patent implements dynamic selection and migration of MTrees based on current system state and requirements. The system can dynamically choose which MTrees to migrate, adjust migration priorities, and coordinate multiple migration operations. This dynamic approach allows the system to maintain fine-grained control while achieving bulk transfer performance when needed.
3Productivity
If bulk filesystem transfer is used, then throughput is increased, but consistency and efficiency are compromised
Solution Approach 1:
The patent takes snapshots of MTrees before migration to establish a consistent starting point. This preliminary action ensures that the migration operates on a known, consistent state of the filesystem, preventing issues related to concurrent modifications. The snapshot mechanism allows the system to capture the exact state of data before transfer begins.
Solution Approach 2:
The patent implements coordination mechanisms that provide feedback during the migration process. Multiple data protection processes communicate their status and coordinate their actions, allowing the system to maintain consistency across bulk transfers. The feedback mechanism enables detection and resolution of conflicts that may arise during concurrent migrations.
Data Source
AI summary
Embodiments for enabling granular migration of data with high efficiency. A defined metadata element, a tag, is assigned to each file, and then tag filtering is used to direct the data to the proper location. Files with different tags can be selected for transfer, and such a group of tags is referred to as a tag set. Embodiments can be used with a defined backup system file migration process, such as present in the Data Domain File System. By using snapshots, incoming new data (ingested file) is allowed to continue while the migration is in process and maintaining data consistency at the same time. This is achieved by performing operations on B+ Tree snapshots in conjunction with tag filtering on keys present in the leaf pages of these structures. This method is efficient became it makes a single pass walk of a B+ Tree in contrast with previous methods that look up files one-by-one via their pathname.