Incremental Inline Journaling for File System Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In journaled file systems, the process of expiring deltas of committed transactions is complex and contributes to tail latency and reduced throughput, particularly due to the delta reaper's operation of holding buf locks and scanning deltas.
Innovation Solution
Implementing incremental inline journaling by collecting journal deltas inline with their application, durably recording a point-in-time consistent value of a delta field, and atomically updating the corresponding predecessor block to free the delta block, providing a single-reap-per-delta semantic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the delta reaper holds buf locks and scans deltas to expire committed transactions, then transaction consistency is maintained, but tail latency increases and throughput decreases
Solution Approach 1:
The patent applies preliminary action by recording a point-in-time consistent value of the delta field inline with the journal entry before the delta reaper operates. This pre-recording of consistent state information allows the system to verify transaction completion without requiring the delta reaper to hold buf locks or scan deltas, thereby reducing tail latency while maintaining transaction consistency through the pre-captured consistent state
2Ease of manufacture
If the delta reaper scans deltas to expire committed transactions, then journal maintenance is performed, but throughput is reduced
Solution Approach 1:
The patent extracts the consistent state information from the delta scanning process by recording it inline in the journal entry. This extraction eliminates the need for the delta reaper to scan deltas to determine transaction completion, allowing journal maintenance to proceed without blocking throughput-critical operations. The consistent state is taken out and stored where it can be efficiently accessed without scanning
Solution Approach 2:
The journal entry structure is enhanced to self-record the point-in-time consistent value of the delta field, making the journal self-sufficient for tracking transaction state. This self-service mechanism eliminates the need for external delta reaper scanning to maintain journal integrity, allowing throughput to improve while journal maintenance continues through the embedded consistent state information
Data Source
AI summary
Incremental inline journaling in a journaled file system can be utilized to facilitate concurrency and throughput of a journaled file system. In an example, a request can be received for a transaction to modify a sub-block portion of a file stored in a distributed storage system. In response to that, the modification can be allocated to a data structure in a journal of the distributed storage system. After the transaction commits, a current value of the sub-block portion of the file can be saved to the delta structure. Then, a new value corresponding to the request for the transaction to modify the sub-block portion of the file can be written to the metadata block. After writing the new value to the metadata block, a memory space used to store the delta structure can be freed.