Implicit Group Commit for Database Log Buffers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database logging systems face performance limitations due to high overhead in I/O operations, leading to restricted transaction per second (TPS) rates and inefficient use of disk space, as each commit incurs synchronous I/O and generates numerous small requests, overwhelming the I/O subsystem.
Innovation Solution
Implementing a group commit mechanism where log buffers are distributed across multiple partitions, allowing accumulation of transactions until full or a threshold of incomplete I/O requests is reached, then sent to the I/O subsystem, thereby reducing the number of I/O operations and optimizing disk usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous I/O is performed for each commit, then data consistency is ensured, but I/O overhead increases and TPS rate decreases
Solution Approach 1:
The patent merges multiple individual commit I/O operations into a single group commit operation. Instead of performing synchronous I/O for each commit, the system accumulates multiple commits in log buffers and performs a single I/O operation to write all accumulated commits to the log partition, thereby reducing I/O overhead while maintaining data consistency through the group commit mechanism
Solution Approach 2:
The system performs preliminary accumulation of commits in memory buffers before executing the I/O operation. Log buffers temporarily store multiple commits waiting to be written, and only when the buffer is full or a threshold is reached does the system perform the actual I/O operation, allowing multiple commits to be prepared in advance and written together
2Ease of operation
If individual commit requests are sent to I/O subsystem, then each commit is processed independently, but the number of I/O operations increases and disk space is wasted
Solution Approach 1:
The patent combines multiple independent commit requests into a single group commit operation. The log buffer structure allows multiple commits to be accumulated and then written together in one I/O operation, reducing the total number of I/O operations while still processing each commit independently in terms of transaction handling and logging
3Speed
If log buffers are sent to I/O subsystem immediately when loaded, then I/O latency is reduced, but the number of incomplete I/O requests increases and system overhead increases
Solution Approach 1:
The system performs preliminary accumulation of commits in log buffers before sending to the I/O subsystem. Instead of immediately sending each commit as it arrives, the system waits until the buffer is full or a threshold condition is met, then performs a batch send operation, reducing the number of I/O requests while managing the buffering process systematically
Data Source
AI summary
First k log buffers, in a data storage application including a plurality of log buffers are each loaded with exactly one transaction. Each log buffer is written to one of a plurality of log partitions. Thereafter, each of the log buffers is sent to an input/output (I/O) subsystem when they are respectively loaded with a single transaction. Transactions are subsequently accumulated in respective new current log buffers after sending the k log buffers to the I/O subsystem. The accumulated transactions are sent to the I/O subsystem when the earlier occurs of (i) the respective current log buffer being full or (ii) a number of incomplete input/output requests handled by the I/O subsystem falls below k. Related apparatus, systems, techniques and articles are also described.