File Write Latency Reduction via Pre-Allocated Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In non-sequential file systems, write input/output operations experience high latency due to the need to read pointer blocks from storage media for cache misses, which is inefficient and slows down first writes.
Innovation Solution
The solution involves pre-allocating blocks for cache misses and journaling the incoming data, allowing for parallel processing to commit the data into the file system, thereby reducing the latency by removing the need to read pointer blocks from storage media.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If pointer blocks are read from storage media for cache misses, then data can be written to the correct location, but write latency increases significantly
Solution Approach 1:
The system pre-allocates storage blocks and prepares write buffers in advance before actual write operations occur. When a write I/O arrives, the data is immediately written to pre-prepared blocks without waiting for pointer block reads, eliminating the latency bottleneck while maintaining correct data placement through subsequent metadata updates.
Solution Approach 2:
The write operation is divided into separate phases: data writing to pre-allocated blocks occurs independently from metadata updates and pointer block reads. This segmentation allows the critical data write path to proceed without waiting for slower metadata operations, improving overall write throughput while maintaining data integrity.
2Loss of time
If all pointer blocks are retained in cache, then write operations are fast, but cache memory requirements become impractical for large files
Solution Approach 1:
The system extracts the pointer block read operation from the critical write path by using pre-allocated blocks. Instead of requiring pointer blocks to be in cache for fast writes, the system writes data directly to pre-prepared blocks and updates metadata separately, reducing cache memory requirements while maintaining write performance.
Solution Approach 2:
Pre-allocated blocks serve as an intermediary between the write I/O and the final file structure. Data is first written to these intermediate blocks without requiring pointer block cache hits, and then metadata is updated to reflect the correct file organization, eliminating the need to keep pointer blocks in cache.
3Measurement precision
If pointer blocks are read sequentially for each write, then data placement accuracy is maintained, but write throughput decreases
Solution Approach 1:
Block allocation and preparation is performed in advance before write operations occur. The system pre-allocates and prepares storage blocks with all necessary metadata structures in place, so that actual write operations can proceed immediately without sequential pointer block reads, maintaining data placement accuracy while improving throughput.
Solution Approach 2:
The system maintains continuous write operations by having pre-allocated blocks ready in advance. Instead of interrupting the write stream to read pointer blocks sequentially, the write operation continues uninterrupted to pre-prepared blocks, with metadata updates occurring in the background to maintain placement accuracy.
Data Source
AI summary
Reducing file write latency includes receiving incoming data, from a data source, for storage in a file and a target storage location for the incoming data, and determining whether the target storage location corresponds to a cache entry. Based on at least the target storage location not corresponding to a cache entry, the incoming data is written to a block pre-allocated for cache misses and the writing of the incoming data to the pre-allocated block is journaled. The writing of the incoming data is acknowledged to the data source. A process executing in parallel with the above commits the incoming data in the pre-allocated block with the file. Using this parallel process to commit the incoming data in the file removes high-latency operations (e.g., reading pointer blocks from the storage media) from a critical input/output path and results in more rapid write acknowledgement.