Log-Structured File System Parallel Append
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel computing environments, concurrent global appending of data to a shared file by multiple distributed processes leads to performance degradation due to serialization, as the parallel file system locks the shared file to maintain consistency, causing processes to wait for each other.
Innovation Solution
Implementing a log-structured file system, such as the Parallel Log-Structured File System (PLFS), which allows non-locking, non-coordinated global appending by storing data with logical offsets that are resolved at read time, enabling multiple write streams to be reconstructed into a single logical file, and deferring the mapping of data chunks until the file is opened by a reader.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a parallel file system locks a shared file to maintain consistency during concurrent appends, then data consistency is improved, but system performance deteriorates due to serialization and idle waiting
Solution Approach 1:
The patent segments the shared file into multiple independent append regions, each with its own metadata structure. Multiple processes can simultaneously append to different regions without requiring global locks, thereby maintaining data consistency while eliminating serialization overhead and improving parallel write performance
Solution Approach 2:
The patent introduces an intermediary log-structured metadata layer that mediates between multiple writing processes and the shared file. This intermediary structure manages append operations through log entries with sequential identifiers, allowing concurrent writes to be ordered and consolidated without direct process-to-process coordination, thus preserving consistency while enabling parallelism
2Reliability
If multiple processes sequentially write to a single shared file, then data consistency is maintained, but data processing bandwidth is wasted and disk space is not preserved
Solution Approach 1:
The patent merges multiple independent append operations from different processes into a single log-structured file format. Each process writes to the log concurrently, and the log structure naturally orders entries through sequential identifiers. This merging approach maintains data consistency while eliminating the need for repeated sequential access patterns, thereby reducing data processing bandwidth consumption and preserving disk space through efficient log compaction
3Manufacturing precision
If deterministic offset calculation is performed during writing, then data positioning precision is improved, but write operation complexity and time increase
Solution Approach 1:
The patent performs preliminary action by pre-allocating sequential log entry identifiers and metadata structures during the write operation itself, rather than calculating deterministic offsets based on final file state. This allows processes to write immediately with predetermined positions in the log, ensuring positioning precision while eliminating the time-consuming offset calculation that would otherwise require knowledge of future writes
Data Source
AI summary
Techniques are provided for globally appending data from a group of distributed processes to a shared file using a log-structured file system. Data generated by a plurality of processes in a parallel computing system are appended to a shared file by storing the data to the shared file using a log-structured file system (such as a Parallel Log-Structured File System (PLFS)); and generating an index entry for the data, the index entry comprising a logical offset entry and a timestamp entry indicating a time of the storage, wherein the logical offset entry is resolved at read time. The logical offset entry can be populated with an append placeholder that is resolved when the shared file is read. At read time, a plurality of the index entries associated with the shared file can be sorted using the timestamp entry to deliver the requested shared file to a requesting application.