Distributed Metadata Server Directory Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file system structures, such as Google File System and Hadoop Distributed File System, face limitations in performance and storage space scalability due to reliance on a single metadata server, leading to inefficiencies as the number of servers increases, particularly in managing metadata distribution and file operations.
Innovation Solution
A method for distributing metadata in units of directories across multiple metadata servers, where user requests are processed by a single server for file operations and distributed between servers for directory operations, along with a block grouping and flushing mechanism to optimize performance and storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If metadata are distributed based on a hashing algorithm across all servers, then storage space is utilized, but directory storage space is wasted and directory-processing performance is degraded
Solution Approach 1:
The patent segments metadata distribution by creating a hierarchical structure where a primary metadata server manages directory-level metadata while secondary metadata servers handle file-level metadata. This segmentation allows directory operations to be concentrated on the primary server, avoiding the performance degradation that occurs when all servers participate in every directory operation.
Solution Approach 2:
The primary metadata server acts as an intermediary between users and secondary metadata servers. It receives directory-related requests, processes them locally, and only forwards file-specific requests to appropriate secondary servers. This intermediary role eliminates the need for all servers to process every directory operation, thereby improving directory-processing performance while maintaining storage distribution.
2Adaptability or versatility
If requests are sent to all data servers, then file operations can be performed, but file creation and deletion performance decreases with an increase in the number of data servers
Solution Approach 1:
The patent segments the metadata server functionality into primary and secondary servers with distinct roles. The primary server handles directory operations (creation, deletion, searching), while secondary servers handle file operations. This segmentation allows file creation and deletion to be processed by specific secondary servers rather than all servers, improving performance as the system scales.
Solution Approach 2:
Different metadata servers are assigned different qualities or responsibilities: the primary server has authority over directory-level operations, while secondary servers have authority over file-level operations within their assigned namespaces. This local quality assignment ensures that file operations are handled by the most appropriate server, improving overall system performance.
3Device complexity
If a single metadata server is used, then the structure is simple, but performance and storage space scalability are limited
Solution Approach 1:
The patent divides the metadata server system into a primary metadata server and multiple secondary metadata servers, each with specific responsibilities. The primary server maintains the directory structure and assigns namespaces, while secondary servers store and manage actual file metadata. This segmentation enables the system to scale horizontally by adding secondary servers without significantly increasing overall system complexity.
Solution Approach 2:
The patent introduces a hierarchical dimension to the metadata server architecture, with the primary server operating at the directory level and secondary servers operating at the file level. This dimensional organization allows the system to scale by adding secondary servers in the file-level dimension while the primary server maintains oversight at the directory level, thereby improving scalability without proportionally increasing complexity.
4Quantity of substance
If metadata space is filled up, then storage capacity is maximized, but metadata must be migrated to another server which drastically degrades performance
Solution Approach 1:
The patent segments metadata storage across multiple secondary servers, each responsible for specific namespaces or portions of the file system. When one secondary server's metadata space is filled, new metadata can be assigned to other secondary servers without migrating data. This segmentation eliminates the performance-degrading migration process while maintaining maximum storage capacity utilization.
Solution Approach 2:
The system dynamically assigns namespaces and metadata storage locations based on current server capacity and load conditions. The primary metadata server can reassign namespaces to different secondary servers as capacity changes, allowing the system to adapt to varying storage conditions without forced migrations and performance degradation.
Data Source
AI summary
Disclosed herein are a metadata server and a method for distributing metadata in units of directories using the metadata server. The method includes receiving a request for at least one of creation, deletion and retrieval from a user file system; acquiring an inode corresponding to an identifier of a parent inode, which is included in the request; searching the parent inode for a dentry corresponding to the request when the request is a request to create a directory; selecting a secondary metadata server in which a new directory is to be created in order to distribute metadata in units of directories; requesting the selected secondary metadata server to create an inode; receiving an identifier of the created inode from the secondary metadata server; and adding a new dentry to the parent inode using the identifier of the inode.