Small-Block Log System for Reducing Write Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In storage systems, high write amplification occurs due to the mismatch between small logical log entries and fixed-size blocks in block-oriented devices, leading to increased latency and wasted storage, especially when handling low OIO (outstanding I/O) scenarios with multiple logical logs.
Innovation Solution
Implementing a small-block log system where write requests are logged in high-speed memory and aggregated into fixed-size blocks before being written to storage, reducing write amplification by batching log entries into 4 KB blocks and using a write-optimized data structure like a log-structured merge tree.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If small logical log entries are written directly to block-oriented storage devices, then data consistency is maintained, but write amplification increases significantly
Solution Approach 1:
The patent segments the write path by introducing a separate log structure that captures write intent at the byte level rather than writing directly to block-oriented storage. This segmentation allows the system to maintain data consistency through detailed logging while avoiding the write amplification problem by decoupling the logging mechanism from the block storage write operations.
Solution Approach 2:
The patent introduces an intermediary log structure that acts as a buffer between the write path and block-oriented storage devices. This intermediary captures write operations in a format optimized for the write path, then reconciles with block storage separately, thereby maintaining data consistency without forcing small log entries through the block storage write path where write amplification occurs.
2Reliability
If small logical log entries are flushed to disk before client acknowledgment, then data durability is ensured, but latency increases
Solution Approach 1:
The patent segments the durability guarantee mechanism by separating the log capture phase from the disk flush phase. The log structure captures write intent immediately and provides durability guarantees through its own persistence mechanisms, while the actual block storage flushes can proceed separately without waiting for client acknowledgment, thereby reducing latency while maintaining data durability.
Solution Approach 2:
The patent performs preliminary logging of write operations in a structure optimized for fast capture and durability tracking before the actual block storage operations complete. This preliminary action in the log structure provides the durability guarantee needed for client acknowledgment without requiring the slower block storage flush to complete first, thereby reducing latency.
3Reliability
If multiple instances of logical logs are maintained, then data consistency across virtual machines is improved, but write amplification increases
Solution Approach 1:
The patent creates a universal log structure that serves multiple virtual machines and logical logs simultaneously. Instead of maintaining separate log instances for each virtual machine that would each write to block storage independently, the system uses a shared log structure that captures writes from multiple sources and reconciles them with block storage in a consolidated manner, thereby maintaining data consistency across VMs while reducing write amplification.
Solution Approach 2:
The patent merges multiple logical log instances into a unified log structure that handles writes from multiple virtual machines. By combining the logging functionality into a single structure that serves multiple purposes and multiple clients, the system maintains data consistency across all VMs while avoiding the multiplicative write amplification that would result from each VM having its own separate log-to-disk write path.
Data Source
AI summary
Writing to a storage system with data striping includes storing blocks of data in local memory until one or more full-stripe write operations can be performed, thus reducing write amplification on the data striped storage system. Crash recovery information includes storing the data associated metadata to respective persistent storage devices. Metadata associated with data from several clients is combined into fixed-size data blocks and stored on the respective persistent storage device.