Parallel Flushing with Bucketized Log Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In storage systems, the single-threaded flushing process of log records to the backing store often fails to keep pace with incoming writes, leading to performance issues due to limitations on Logical Block Address (LBA) range overlapping and job workloads, which can result in user I/O suffering from reduced performance.
Innovation Solution
Organizing log records into buckets associated with LBA ranges and selecting buckets with the lowest Log Sequence Numbers (LSNs) for parallel flush jobs, allowing simultaneous flushing to the backing store while maintaining LBA range integrity and rebalancing the tree as necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single-threaded flush process is used, then the system is simple to implement, but the flush rate is insufficient to keep pace with incoming writes
Solution Approach 1:
The flush process is segmented by dividing log records into multiple buckets based on LBA ranges, and then selecting buckets for parallel flush jobs. This segmentation allows the single-threaded flush process to be transformed into a multi-parallel-flush-process, increasing productivity without requiring complete redesign of the entire flushing mechanism.
Solution Approach 2:
The patent introduces a new dimension of parallelism by organizing flush operations across multiple buckets with different LBA ranges simultaneously. This dimensional approach to parallelization enables the system to flush records in parallel without requiring complex coordination mechanisms, thus improving flush rate while maintaining manageable complexity.
2Productivity
If multiple parallel flush jobs are submitted, then the flush rate increases, but the backing store has limitations on LBA range overlapping and job workload
Solution Approach 1:
Each bucket is assigned specific LBA ranges, and flush jobs are constructed by selecting buckets with non-overlapping LBA ranges. This local quality approach ensures that each parallel flush job operates on distinct LBA ranges, satisfying the backing store's constraints while maximizing parallel throughput.
Solution Approach 2:
The system performs preliminary organization of log records into buckets with assigned LBA ranges before initiating flush operations. This preliminary bucketization and selection process ensures that when parallel flush jobs are submitted, they automatically satisfy the backing store's LBA range constraints, avoiding conflicts and ensuring reliable operation.
3Productivity
If log records are flushed sequentially, then LBA range integrity is maintained, but the flush rate cannot keep up with high write rates
Solution Approach 1:
The sequential flush process is segmented into multiple parallel flush jobs, each handling specific LBA ranges. This segmentation maintains LBA range integrity by ensuring that each job operates on well-defined, non-overlapping ranges while improving overall flush rate through parallel execution.
Solution Approach 2:
The system performs preliminary organization of log records into buckets with assigned LBA ranges before flushing. This preliminary action establishes clear boundaries for each parallel job, ensuring that LBA range integrity is maintained even when multiple flush operations execute simultaneously.
Data Source
AI summary
A method, computer program product, and computer system for organizing a plurality of log records into a plurality of buckets, wherein each bucket is associated with a range of a plurality of ranges within a backing store. A bucket of the plurality of buckets from which a portion of the log records of the plurality of log records are to be flushed may be selected. The portion of the log records may be organized into parallel flush jobs. The portion of the log records may be flushed to the backing store in parallel.