Parallel File System Metadata Distribution via MDHIM Sharding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel file systems, the large amount of metadata required for shared files leads to performance degradation and resource competition when multiple processes write concurrently, as all processes must redundantly store and process extensive metadata, causing serialization and resource contention.
Innovation Solution
Implementing a partitioned data store with Multidimensional Data Hashing Indexing Middleware (MDHIM) and low-latency Message Passing Interface (MPI) communications across compute nodes to shard and distribute metadata, allowing only necessary metadata portions to be stored and processed, reducing data processing and transfer bandwidth costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all processes store and process the entire metadata set for shared files, then data consistency is maintained, but system performance degrades and resource contention increases
Solution Approach 1:
The patent segments the metadata for a shared file into multiple partitions, with each partition stored on a different compute node. This allows processes to access only the metadata partitions relevant to their operations rather than requiring all processes to load and process the entire metadata set, thereby improving write performance while maintaining data consistency through distributed metadata management.
2Productivity
If metadata is distributed across partitioned data stores, then resource utilization improves and concurrency increases, but system complexity increases
Solution Approach 1:
The patent introduces a metadata management system that acts as an intermediary layer between processes and the distributed metadata partitions. This mediator handles the complexity of metadata distribution, partition routing, and consistency management, allowing processes to interact with a simplified interface while the underlying distributed system manages the complexity of metadata partitioning across multiple compute nodes.
3Reliability
If all reading processes read all metadata created by all writing processes, then data integrity is ensured, but bandwidth costs and processing overhead increase
Solution Approach 1:
The patent implements local quality by allowing reading processes to access only the specific metadata partitions relevant to their read operations rather than requiring all reading processes to read and process the entire metadata set. Each compute node stores and manages specific partitions of the metadata, enabling localized access that maintains data integrity for the requested data while reducing unnecessary bandwidth consumption and processing overhead.
Data Source
AI summary
Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).