Lockless RDMA Buffer Allocation via Pre-Registered Memory Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High latency and increased CPU consumption due to lock acquisition and release in RDMA buffer pools, particularly in systems performing a large number of I/O operations.
Innovation Solution
Maintaining a global pool of pre-registered RDMA buffers that can be locklessly obtained by individual threads, using techniques such as static and dynamic chunk-to-thread mapping to manage memory allocation and caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locking mechanisms are used to maintain free lists of buffers in RDMA buffer pools, then buffer reuse safety is ensured in multi-process systems, but system latency increases and performance degrades due to lock acquisition and release overhead
Solution Approach 1:
The buffer pool is divided into multiple per-thread buffer pools, with each thread having its own dedicated free list. This segmentation eliminates the need for locking when threads access their own buffer pools, as each thread operates independently on its own data structures. The division of the global buffer pool into thread-local segments resolves the contradiction by ensuring buffer reuse safety through isolation while eliminating lock acquisition overhead.
Solution Approach 2:
Each thread manages its own buffer pool and free list independently without requiring external coordination or locking mechanisms. Threads self-service their buffer allocation and reuse needs by operating on their own thread-local data structures, which eliminates the performance penalty of lock acquisition and release while maintaining safety through isolation.
2Reliability
If locking mechanisms are used for buffer management in RDMA operations, then concurrent access safety is maintained, but CPU consumption increases due to lock overhead
Solution Approach 1:
The buffer management system is segmented into thread-local components, where each thread has its own free list and buffer pool. This segmentation eliminates the need for locking during concurrent access, as threads operate independently on their own data structures. The segmentation approach maintains concurrent access safety through isolation while eliminating the CPU overhead associated with lock acquisition and release.
Solution Approach 2:
Threads independently manage their own buffer pools without requiring coordination with other threads through locking mechanisms. This self-service approach reduces CPU consumption by eliminating lock overhead while maintaining safety through thread-local isolation of buffer management operations.
3Productivity
If memory buffers are registered with RDMA NIC for each I/O operation, then RDMA transfers can be performed, but registration costs accumulate and degrade system performance when I/O operation counts are high
Solution Approach 1:
Memory buffers are pre-registered with the RDMA NIC before I/O operations begin, rather than being registered at the time of each I/O operation. This preliminary action allows buffers to be registered once and reused across multiple I/O operations, eliminating the cumulative registration cost that degrades performance in high I/O count scenarios. The pre-registration maintains RDMA transfer capability while significantly reducing the time lost to registration operations.
Solution Approach 2:
Instead of registering buffers for each I/O operation and losing the registration state, the system recovers and reuses the same registered buffers across multiple I/O operations. Buffers remain registered with the RDMA NIC and are reused until they are actually freed, eliminating redundant registration costs while maintaining the ability to perform RDMA transfers.
Data Source
AI summary
Methods, systems and computer program products for lockless acquisition of memory for RDMA operations. A contiguous physical memory region is allocated. The contiguous physical memory region is divided into a plurality of preregistered chunks that are assigned to one or more process threads that are associated with an RDMA NIC. When responding to a request from a particular one of the one or more process threads, a buffer carved from the preregistered chunk of the contiguous physical memory region is assigned to the requesting process thread. Since the memory is pre-registered, and since the associations are made at the thread level, there is no need for locks when acquiring a buffer. Furthermore, since the memory is pre-registered, the threads do not incur registration latency. The contiguous physical memory region can be a contiguous HugePage contiguous region from which a plurality of individually allocatable buffers can be assigned to different threads.