Parallel In-Memory Database Page Flushing via Chunked I/O
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of flushing pages into physical persistence in in-memory databases can be resource intensive and slow down database performance, particularly due to slow write I/O operations when handling numerous small pages.
Innovation Solution
Assigning physical block numbers to pages and distributing them to helper threads for parallel processing, where pages with consecutive block numbers are combined into chunks and flushed together, and pages with non-consecutive block numbers are distributed to different threads, allowing for efficient write I/O operations by filling small page buffers before flushing to disk persistence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pages are flushed individually into physical persistence, then data durability is ensured, but write I/O efficiency deteriorates and resource consumption increases
Solution Approach 1:
The patent combines multiple small pages into larger chunks before flushing to disk. Specifically, pages are grouped into chunks where each chunk contains multiple pages, and these chunks are then flushed together in single I/O operations. This merging approach maintains data durability while significantly improving write I/O efficiency by reducing the number of separate I/O operations required.
Solution Approach 2:
The patent performs preliminary actions by pre-allocating chunk structures and pre-grouping pages into chunks before the actual flush operation. The resource flush thread prepares chunks in advance by collecting pages that will be flushed together, so that when flushing is needed, the pages are already organized for efficient batch writing to disk.
2Productivity
If multiple helper threads are used for parallel page processing, then processing speed is improved, but system complexity increases
Solution Approach 1:
The patent divides the page flushing workload into multiple independent helper threads, each responsible for processing specific pages or chunks. This segmentation allows parallel processing of page flushing operations, improving overall processing speed while keeping each individual thread relatively simple in structure and function.
Solution Approach 2:
The patent introduces a resource flush thread as an intermediary that coordinates between the helper threads and the actual disk I/O operations. This mediator thread manages the chunk formation, monitors when chunks are full, and triggers the actual flush operations, thereby simplifying the complexity management by centralizing coordination logic in a single intermediary component.
Data Source
AI summary
Physical block numbers are assigned to pages loaded into memory of an in-memory database. Thereafter, pages having a same size are distributed to one of a plurality of helper threads executing in parallel for processing if such pages having consecutive physical block numbers. Each of these helper threads can have a corresponding small page buffer. Contents of the respective small page buffers are later flushed to physical disk persistence when such small page buffers are full.