Striped Metadata Storage in Parallel File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel file systems, the large amount of metadata required for concurrent writing processes leads to performance degradation and excessive storage needs, as all writing processes must redundantly store and read the same metadata, causing bandwidth and disk space issues.
Innovation Solution
Metadata is striped across multiple subdirectories in a round-robin manner and stored on object storage servers, allowing only the necessary metadata to be accessed during read operations, reducing metadata lookup time and memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is stored centrally for all sub-files, then data consistency is maintained, but read performance degrades due to large metadata footprint and bandwidth consumption
Solution Approach 1:
The patent segments metadata into two types: common metadata (file-level information applicable to all sub-files) and sub-file metadata (individual sub-file information). Common metadata is stored once and shared across all sub-files, while sub-file metadata is stored separately. This segmentation reduces redundant metadata storage and improves read performance by allowing processes to access only relevant sub-file metadata without loading entire file metadata into memory.
2Reliability
If all writing processes store the same metadata redundantly, then data consistency is ensured, but disk space is wasted and bandwidth is consumed
Solution Approach 1:
The patent extracts common metadata from individual sub-file metadata structures and stores it separately at the file level. This extraction eliminates redundant storage of identical metadata across multiple sub-files. When a process needs to access metadata, it retrieves the shared common metadata once and combines it with specific sub-file metadata, significantly reducing disk space consumption and network bandwidth usage compared to storing complete metadata copies for each sub-file.
3Speed
If metadata is read into memory for processing, then access speed improves, but memory footprint increases significantly
Solution Approach 1:
The patent segments metadata into common and sub-file portions, allowing processes to load only the necessary sub-file metadata into memory while keeping common metadata in a more compact shared format. This segmentation enables selective loading of metadata based on process needs, reducing memory footprint while maintaining fast access speeds for frequently accessed metadata through efficient data structures and caching strategies.
Data Source
AI summary
Metadata associated with a plurality of sub-files associated with a single shared file is stored in a parallel file system. A plurality of processes generate a shared file. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file and metadata for the at least one portion of the shared file on one or more of the plurality of object storage servers. The compute node is further configured to store the metadata by striping the metadata across a plurality of subdirectories of the shared file. The metadata is optionally striped across the plurality of subdirectories in a round-robin manner. The plurality of subdirectories are stored on one or more of the object storage servers. Write and read processes optionally communicate using a message passing interface. A given write process optionally writes metadata for a given portion of the shared file to an index file in a particular one of the subdirectories corresponding to the given portion.