Distributed File System Metadata Allocation via Token-Based Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computer systems, the co-location of file data and metadata on a single storage node leads to scalability issues as the number of files grows, affecting system efficiency and reliability.
Innovation Solution
File metadata is stored separately from file data and distributed among a plurality of metadata nodes, allowing for improved scalability by decoupling file index nodes from metadata nodes, with token-based policies for efficient directory and file placement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If file data and metadata are co-located on a single storage node, then system simplicity is maintained, but scalability deteriorates as the number of files grows
Solution Approach 1:
The patent segments the storage system into separate metadata nodes and data nodes. Metadata is extracted from storage nodes and distributed across dedicated metadata nodes, while data remains on data nodes. This segmentation resolves the contradiction by allowing the system to maintain simplicity in individual node design while achieving scalability through distributed architecture.
Solution Approach 2:
The patent extracts metadata from storage nodes and places it on separate metadata nodes. This extraction allows the storage nodes to focus solely on data operations while metadata operations are handled by specialized metadata nodes, improving overall system scalability without increasing individual node complexity.
2Productivity
If file metadata is stored separately from file data on multiple metadata nodes, then system scalability is improved, but system complexity increases
Solution Approach 1:
The patent merges the metadata storage function across multiple metadata nodes into a unified namespace. The distributed metadata namespace provides a unified view of metadata across all metadata nodes, allowing the system to achieve scalability while maintaining operational simplicity through a unified interface.
Solution Approach 2:
The patent creates a universal metadata namespace that can be accessed by any data node or client through a standardized interface. This universal namespace allows different metadata nodes to work together seamlessly, providing scalability without increasing the complexity of individual operations.
3Productivity
If metadata is distributed among multiple nodes, then retrieval efficiency is improved through parallel access, but network overhead increases
Solution Approach 1:
The patent implements preliminary actions by pre-computing and storing metadata location information in the distributed metadata namespace. When metadata is needed, the system can quickly locate the relevant metadata node through the namespace without scanning the entire system, reducing network overhead while maintaining parallel access efficiency.
Data Source
AI summary
Systems and methods for allocating file system metadata to storage nodes of a distributed file system. An example method may include: defining, by a processing device, a plurality of tokens, wherein each token comprises a sequence of bits; associating each token of the plurality of tokens with a metadata node of a cluster of metadata nodes; receiving a request to create a directory; selecting, among the plurality of tokens, a token associated with the directory; generating a directory identifier comprising the selected token; and creating, on a metadata node associated with the selected token, a directory record comprising the directory identifier.