Leader Thread Batching for Shared Log Buffer Contention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face performance degradation due to high contention for shared log buffer space in larger servers, particularly in high-core multi-NUMA systems, where lock-free logging systems fail to optimize thread synchronization effectively.
Innovation Solution
A lockless transaction log synchronization method where a leader thread is elected to reserve and copy its own transaction log, allowing other threads to attach and then reserve space for member threads, reducing contention by having a single thread manage multiple concurrent threads' logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If lock-free logging is used to permit concurrent copying of transaction logs onto the shared log buffer, then thread synchronization is optimized and database parallelism is improved, but contention for shared log buffer space increases and performance degrades in larger servers
Solution Approach 1:
The system segments threads into batches with elected leader threads. Each leader thread manages a batch of member threads, dividing the concurrent access problem into smaller, manageable groups. This segmentation reduces contention by limiting the scope of coordination to batch-sized threads rather than all threads simultaneously.
Solution Approach 2:
Leader threads perform preliminary actions by reserving log buffer space in advance for their entire batch of member threads before any actual logging occurs. This preliminary reservation prevents contention during the actual logging phase, as buffer space is already allocated and ready for batched writing operations.
2Productivity
If the number of transaction threads is increased to handle more client requests, then database throughput is improved, but contention for shared log buffer space is worsened
Solution Approach 1:
The system merges multiple member threads into batches under a single leader thread. These batched threads collectively reserve and write to the log buffer as a unified group, reducing the overhead of individual thread coordination. This merging allows more threads to operate concurrently by organizing them into efficient batch units.
Solution Approach 2:
Leader threads act as intermediaries between member threads and the shared log buffer. Instead of each thread directly contending for buffer space, the leader thread mediates by reserving space for the entire batch and coordinating the writing process, thereby reducing direct contention among numerous threads.
3Reliability
If traditional locking mechanisms are used to regulate shared log buffer access, then thread synchronization is ensured, but database parallelism is reduced and scalability is limited
Solution Approach 1:
Leader threads self-organize batches of member threads and autonomously manage buffer space reservation and coordination without requiring external locking mechanisms. This self-service approach maintains synchronization reliability while avoiding the parallelism limitations of traditional locks, as threads serve themselves through batched coordination.
Data Source
AI summary
Methods and system for a database management system (DBMS) in which a leader thread is elected from concurrent transaction threads stored in one or more data nodes. While the leader thread copies its own thread transaction log onto a reserved portion of the shared log buffer, the leader thread permits other transaction threads to attach to a thread chain starting with the leader thread. Once the leader has completed copying its thread transaction log onto the shared log buffer, it then reserves a portion of the shared log buffer, and copies the member thread transaction logs onto the shared log buffer to reduce the contention for shared buffer may be reduced.