Adaptive Redo Log Writer Groups for Lower Commit Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing write-ahead logging (WAL) systems introduce latency during transaction commit due to the need for log file sync, which limits online transaction processing (OLTP) throughput as redo log records accumulate in the log buffer before being persisted.
Innovation Solution
The system employs multiple modes of operation - thin mode and thick mode - to adaptively overlap redo writes. In thin mode, small redo writes are distributed across a large number of log writer groups, while in thick mode, larger redo writes are handled by a smaller number of groups, allowing the system to better handle varying loads and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple log writer groups are used to overlap redo writes, then throughput is improved, but device complexity increases
Solution Approach 1:
The log writer is divided into multiple log writer groups (first log writer group and second log writer group), each capable of independently writing redo log records to persistent storage. This segmentation enables parallel processing of redo writes, improving throughput while managing complexity through modular organization of writing tasks across different storage devices or storage locations.
2Adaptability or versatility
If a fixed number of log writer groups are used, then device complexity is reduced, but adaptability to varying loads deteriorates
Solution Approach 1:
The system dynamically selects between a first log writer group and a second log writer group based on current load conditions and performance characteristics. The log writer can adaptively switch between using one or two groups depending on the situation, allowing the system to optimize for both throughput and latency without requiring a fixed large number of groups, thus managing complexity while improving adaptability.
3Reliability
If redo writes are accumulated in the log buffer, then durability is ensured, but latency increases
Solution Approach 1:
The system performs preliminary actions by maintaining multiple log writer groups ready to write redo records. When redo log records are generated, they can be immediately written by an available log writer group to persistent storage without significant accumulation in the log buffer. This preliminary preparation of writing capacity reduces the waiting time (latency) while still ensuring durability through confirmed writes to persistent storage.
Solution Approach 2:
By having multiple log writer groups capable of operating concurrently or in sequence, the system ensures continuous writing of redo log records to persistent storage. This continuity reduces gaps where records would accumulate in the buffer waiting for writing, thereby reducing latency while maintaining the durability guarantee that redo records are persisted before transaction commit completes.
Data Source
AI summary
The present disclosure relates to adaptively overlapping redo writes. A log writer, while operating in a thin mode, may assign a first log writer group of a plurality of log writer groups to write one or more first redo log records to an online redo log in response to determining that a pipelining parameter is satisfied. The thin mode may be associated with one or more target sizes that are less than one or more target sizes associated with a thick mode. The log writer may determine to operate the thick mode based at least in part on at least a portion of the plurality of log writer groups being unavailable to write one or more second redo log records to the online redo log. The log writer, while operating in the thick mode, may assign a second log writer group of the plurality of log writer groups to write one or more second redo log records from the log buffer to the online redo log in response to determining that an amount of redo log records in the log buffer meets one of the one or more target sizes associated with the thick mode. The log writer, while operating in the thick mode, may assign a third log writer group of the plurality of log writer groups to write one or more second redo log records from the log buffer to the online redo log in response to determining that a highest busy group number meets or exceeds a core threshold.