Shared Send Queue Pool for RDMA Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High memory footprint in reliable connection (RC) transport services using InfiniBand protocol and RDMA over Converged Ethernet (ROCE) limits the scalability of services to handle a large cluster of computing devices.
Innovation Solution
Implementing a scalable RC (SRC) system that reduces memory footprint by using a pool of shared send queues (SSQs), where each SSQ can be used by multiple queue pairs (QPs), and allocating SSQs dynamically based on demand.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If individual send queues are allocated for each queue pair (QP) in RC transport, then connection reliability is maintained, but memory footprint increases
Solution Approach 1:
Multiple queue pairs (QPs) share a common send queue (SQ) resource instead of each QP having a dedicated SQ. The system merges the SQ functionality across multiple QPs while maintaining individual QP identities for connection management. This reduces total memory usage while preserving connection reliability through the shared queue mechanism.
Solution Approach 2:
A single send queue (SQ) is designed to serve multiple queue pairs (QPs) simultaneously. The SQ becomes a universal resource that handles sending operations for multiple connections, replacing the traditional one-to-one mapping between QP and SQ. This multi-functionality reduces the total number of queues needed while maintaining RC features.
2Ease of operation
If individual send queues are allocated for each queue pair (QP), then connection management is simplified, but scalability to large clusters is limited
Solution Approach 1:
The system merges multiple QP-specific send queues into a shared send queue resource pool. This consolidation reduces the overhead of managing numerous individual queues while maintaining the ability to manage multiple connections through the shared resource, thereby improving scalability to large clusters.
Solution Approach 2:
The system dynamically allocates and manages shared send queues among multiple queue pairs based on runtime conditions. Instead of static one-to-one allocation, the shared queue resources can be dynamically assigned to different QPs as needed, enabling the system to adapt to varying connection demands and scale efficiently.
Data Source
AI summary
In some implementations, the system may include one or more processors, coupled to memory, the processor configured to set in the memory a pool of shared send queues (SSQs), each SSQ in the pool of SSQs is configured for use as a send queue (SQ) for one or more queue pairs (QPs). The one or more processors may allocate at least one of SSQs from the pool of SSQs to the QP, the SSQ set for a process having a plurality of connections to a plurality of remote processes. The one or more processors may send by the SSQ, via the plurality of connections, outgoing messages to separate remote processes of the plurality of remote processes. Also, the system may include implementations where the pool is set responsive to a first number of the QPs reaching a threshold. The system may include implementations where the pool of SSQs is pinned in the memory.