Parallel Log-Structured File System for Small File Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Parallel storage systems face performance degradation due to serialization and non-sequential disk access patterns when multiple processes concurrently write small files to a shared storage system, leading to increased latency and reduced efficiency.
Innovation Solution
Implementing a Parallel Log-Structured File System (PLFS) as a middleware virtual file system to aggregate multiple small files into a single large file, using metadata to manage the layout and facilitate sequential writes, thereby reducing overhead and improving disk access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple processes concurrently create small files in a shared directory, then each process can independently write data, but serialization of directory locking causes significant performance degradation
Solution Approach 1:
The patent merges multiple small files into a single aggregate file, allowing multiple processes to write concurrently without directory locking serialization. Instead of creating separate files that require coordinated locking, processes write to different regions of the same file using metadata-guided offsets, eliminating the bottleneck while maintaining data organization.
Solution Approach 2:
The patent introduces a metadata structure as an intermediary between the file system and stored data. This metadata layer manages file offsets, lengths, and aggregation information, allowing concurrent writes without direct file system locking. The metadata acts as a mediator that coordinates access without requiring serialization of the underlying storage operations.
2Productivity
If multiple small files are concurrently written to disk, then data from multiple processes can be stored, but non-sequential access patterns cause frequent disk seeks and performance degradation
Solution Approach 1:
The patent performs preliminary organization of data into an aggregate file structure before writing to disk. By pre-calculating offsets and arranging data in sequential regions within the aggregate file, the system prepares the data layout in advance, eliminating the need for frequent disk seeks during the writing process and improving overall storage throughput.
3Speed
If each process creates a single file with sequential writes, then disk access efficiency is improved, but managing multiple processes' data in separate files increases metadata overhead and disk space consumption
Solution Approach 1:
The patent combines multiple process data streams into a single aggregate file, reducing the number of file metadata structures that need to be managed. Instead of maintaining separate file headers, inodes, and directory entries for each process, the system uses a single aggregate file with an organized internal structure, simplifying metadata management while preserving data separation through offset tracking.
Data Source
AI summary
Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.