InfiniBand Single Receive Queue Interleaving RDMA Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
InfiniBand systems face inefficiencies due to dedicated FIFO queues for each queue pair and virtual lane, leading to wasted memory space and delayed communication transfers when interfacing with external systems, as well as challenges in interleaving different types of traffic and tracking RDMA Read responses.
Innovation Solution
Implementing a shared memory structure using linked lists to buffer InfiniBand commands, allowing for dynamic allocation and reconfiguration of memory buckets, enabling efficient flow control and interleaving of Send commands and RDMA Read responses within a single receive queue, thereby avoiding out-of-order processing and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dedicated FIFO queues are used for each queue pair and virtual lane, then flow control and traffic isolation are ensured, but memory space is wasted and buffer utilization is inefficient
Solution Approach 1:
The patent merges multiple dedicated FIFO queues into a single shared queue structure that can handle traffic from multiple queue pairs and virtual lanes. The shared queue uses a circular buffer with head and tail pointers that can be independently managed for different QPs/VLs, allowing memory space to be shared while maintaining logical separation for flow control purposes.
Solution Approach 2:
The shared queue structure serves multiple functions: it acts as a common buffer for multiple QPs and VLs, provides flow control for each QP/VL independently through separate credit tracking, and enables efficient memory utilization by allowing dynamic allocation of buffer space based on actual traffic needs rather than static worst-case allocations.
2Reliability
If dedicated queues are used for each queue pair, then traffic isolation is maintained, but transfer delays occur when copying traffic between InfiniBand queues and external system queues
Solution Approach 1:
The patent combines the InfiniBand receive queue with the external system transmit queue into a single shared memory structure. This eliminates the need to copy data between separate queues, as both InfiniBand packets and external system frames can be stored in the same memory space with appropriate formatting and pointers, significantly reducing transfer latency while maintaining logical traffic isolation through queue-specific tracking structures.
3Quantity of substance
If a single shared receive queue is used for mixed traffic types, then memory utilization is improved, but flow control becomes problematic due to difficulty in tracking storage usage per queue pair and virtual lane
Solution Approach 1:
The patent segments the shared queue management into independent credit tracking structures for each queue pair and virtual lane. Each QP and VL has its own credit counter that independently tracks the amount of buffer space allocated to it, allowing the shared physical memory to be logically divided into separate flow-controlled channels. This segmentation enables efficient memory utilization while maintaining simple flow control logic for each QP/VL.
4Stability of the object's composition
If separate FIFO queues are used for Sends and RDMA Read responses, then traffic ordering is maintained, but the system cannot efficiently handle interleaved traffic types in a single queue
Solution Approach 1:
The patent implements a dynamic queue structure that can adaptively handle different traffic types (Sends and RDMA Read responses) within a single shared queue. The queue uses type identification fields and dynamic pointer management to distinguish between different traffic types, allowing them to be interleaved in the same physical queue while maintaining proper ordering through type-specific processing logic and separate head/tail pointers for each traffic type when needed.
Data Source
AI summary
A system and method for processing interleaved Sends of encapsulated communications and responses to RDMA Reads in a single InfiniBand queue pair receive queue. The queue is implemented as one or more linked lists of memory buckets, and stores Send commands (containing encapsulated communications or RDMA Read descriptors for retrieving a communication) until their associated communications are assembled and forwarded to a transmit module. The queue grows as new InfiniBand packets are received, and shrinks as communications (e.g., Ethernet packets) are forwarded. A next packet pointer identifies the next Send command whose communication should be assembled. If it is an encapsulated communication, the communication is forwarded. Otherwise, RDMA Read requests are issued and the responses bypass the tail of the queue and are assembled in an assembly area at the head of the queue.