Log-Based Storage Coalescing Write I/Os
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems in cloud environments incur high costs due to the large number of I/O operations required for logging client write I/O operations, particularly when using block level storage volumes optimized for I/O intensive workloads, leading to increased IOPS charges.
Innovation Solution
The implementation of a log-based system that coalesces multiple journaling operations and performs single buffer writes of aggregated data, reducing the number of I/O operations by integrating write I/Os into contiguous segments and persisting them only when the size exceeds a maximum aggregation size or the elapsed time exceeds a specified limit, thereby minimizing IOPS and associated costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple individual write I/O operations are logged to the log, then data integrity and consistency are maintained, but the number of I/O operations increases significantly, leading to high IOPS charges
Solution Approach 1:
The patent merges multiple individual write I/O logging operations into a single bulk logging operation by coalescing journaling operations and performing single buffer writes of aggregated data. This reduces the number of I/O operations to the log while maintaining data integrity through the use of an aggregation queue that accumulates multiple write requests before persisting them together.
Solution Approach 2:
The patent performs preliminary aggregation of write I/O operations in an aggregation queue before actual logging to the log. By pre-coalescing multiple journaling operations and preparing aggregated data in memory, the system reduces the frequency of I/O operations to the log, thereby lowering IOPS charges while ensuring data is ready for reliable persistence when needed.
2Productivity
If write I/Os are aggregated into contiguous segments and persisted in bulk, then the number of I/O operations is reduced, but data persistence timing and availability may be delayed
Solution Approach 1:
The patent implements dynamic aggregation with configurable thresholds for both size and time. The aggregation queue monitors multiple conditions and persists data when either the maximum aggregation size is reached or the aggregation time limit is exceeded. This dynamic approach balances I/O reduction with timely data persistence, allowing the system to adapt to different workload requirements and prevent excessive delays.
3Productivity
If the aggregation queue waits for maximum size or time limit before persisting, then I/O operations are minimized, but the system complexity increases
Solution Approach 1:
The patent segments the logging function into distinct components: an aggregation queue for coalescing operations, integration logic for merging contiguous segments, and persistence logic for bulk writing. This segmentation allows each component to handle specific tasks efficiently while maintaining overall system manageability despite the increased functionality.
Solution Approach 2:
The aggregation queue serves multiple functions: it acts as a buffer for incoming write I/Os, performs coalescing of journaling operations, manages contiguous segment integration, and triggers bulk persistence based on size or time conditions. This multi-functionality reduces the need for separate dedicated components, managing system complexity while achieving I/O reduction goals.
Data Source
AI summary
Processing can be performed to persistently record, in a log, a write I/O that writes first data to a target logical address. The processing can include: allocating storage for a first page buffer (PB) located at offsets in a PB pool of non-volatile storage of the log; enqueuing a request to an aggregation queue to persistently store the first data to the first PB of the log, wherein the request identifies the offsets of the PB pool of non-volatile storage which correspond to the first PB; and integrating the request into the aggregation queue. Integrating can include: determining whether a contiguous segment of the offsets of the request is adjacent to a second contiguous segment of the aggregation queue; and responsive to determining the contiguous segment is adjacent to the second contiguous segment, merging the first and second contiguous segments and generating an aggregated continuous segment.