Metadata Server Directory Migration for Distributed Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing load balancing methods in distributed file systems cause high internal memory consumption and request delays due to large directory migrations, which affect performance and user experience.
Innovation Solution
Create a second directory with a lower hierarchy level for a first directory and migrate it to a target metadata server that meets a preset load condition, avoiding data freezing and reducing the amount of data migration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If directory migration is performed to balance load on metadata servers, then load distribution is improved, but internal memory consumption increases due to large data amounts
Solution Approach 1:
The patent segments the directory migration process by creating a second directory with only the directory entry (metadata) at a lower hierarchy level, separate from the full directory data. This allows migrating minimal data (just the entry) to balance load, while the actual directory contents remain distributed across storage devices, thus reducing internal memory consumption during migration.
Solution Approach 2:
The patent extracts only the essential directory entry (metadata) from the complete directory structure for migration purposes. By taking out just the necessary component (the directory entry at lower hierarchy level) rather than migrating the entire directory data, the system achieves load balancing with minimal memory overhead.
2Reliability
If directory freezing is applied during migration to ensure data consistency, then data integrity is improved, but request processing time increases due to suspended operations
Solution Approach 1:
The patent creates a second directory as a copy of the original directory entry at a lower hierarchy level. This copy can be migrated independently without freezing the original directory, as it represents a snapshot that can be safely replicated. The copying approach ensures data consistency while allowing continuous access to the original directory during migration.
Solution Approach 2:
The patent performs preliminary action by creating the second directory entry before migration, establishing a safe copy that can be moved without affecting the original. This preliminary creation of the migrated copy allows the system to prepare for migration without suspending operations on the source directory.
3Productivity
If complete directory migration is performed to achieve load balancing, then load distribution is improved, but feedback time increases due to large data transfer
Solution Approach 1:
The patent segments the migration task by transferring only the directory entry (metadata) rather than the complete directory data. This segmentation reduces the data transfer volume from potentially gigabytes to minimal metadata size, dramatically reducing feedback time while still achieving load balancing through proper directory entry placement.
Solution Approach 2:
The patent extracts only the essential directory entry information needed for load balancing decisions, leaving the bulk directory contents on their original storage devices. This extraction approach minimizes data transfer time and feedback delays while maintaining the ability to balance load effectively through intelligent directory entry routing.
Data Source
AI summary
Load balancing methods and systems are described herein. A method may be performed by one metadata server in a distributed file system, and may include creating at least one second directory for a first directory in response to a directory creation operation triggered for the first directory. The following operations may be performed on each second directory: use, from other metadata servers, another metadata server whose current load value satisfies a preset load condition as a target metadata server, when it is determined, based on a directory category of the first directory and a hierarchy relationship between the first directory and the second directory, that the second directory satisfies a migration condition; and send directory information of the second directory to the target metadata server, to enable the target metadata server to create the second directory based on the received directory information of the second directory.