InfiniBand RDMA Read Response Tracking via Shared Linked Lists
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
InfiniBand systems face inefficiencies due to dedicated memory queues for each queue pair and virtual lane, leading to wasted memory space and delayed communication transfers between InfiniBand and external systems, as well as challenges in tracking responses to RDMA Read requests.
Innovation Solution
Implementing a shared memory structure using linked lists for multiple queue pairs and virtual lanes, allowing dynamic allocation and reconfiguration, and maintaining linked lists for tracking RDMA Read responses to efficiently manage memory and facilitate flow control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dedicated FIFO queues are provided for each queue pair and virtual lane, then flow control and traffic management are improved, but memory space usage increases due to worst-case buffering requirements
Solution Approach 1:
The patent merges multiple dedicated FIFO queues into a single shared receive queue that handles traffic from multiple queue pairs and virtual lanes. This consolidation eliminates the need for separate worst-case buffering for each QP/VL while maintaining flow control through software-based tracking mechanisms that monitor buffer usage per QP/VL within the shared structure.
Solution Approach 2:
The shared receive queue serves multiple functions simultaneously - it handles traffic from different queue pairs, supports multiple virtual lanes, and provides flow control for all of them through a unified structure. This multi-functional design replaces the need for multiple specialized dedicated queues.
2Ease of operation
If discrete FIFO queues are used for each terminating QP, then traffic management is simplified, but transfer efficiency decreases due to additional copying operations
Solution Approach 1:
The patent combines the InfiniBand receive queue with the external system transmit queue into a single shared queue structure. This eliminates the intermediate copying step where traffic would normally be copied from InfiniBand QP queue to a separate external system queue, thereby improving transfer efficiency while maintaining manageable traffic control through the unified structure.
3Quantity of substance
If shared storage space is used for virtual lane and queue pair traffic, then memory space efficiency is improved, but flow control becomes problematic
Solution Approach 1:
The patent implements flow control in the shared queue environment by having the receive module track buffer usage for each queue pair and virtual lane within the shared structure. When a QP/VL exhausts its allocated buffer space, the system generates feedback signals (RNR-NAK packets) to inform the transmitter to slow down or pause, thereby maintaining flow control despite the shared memory architecture.
Data Source
AI summary
A system and method for tracking responses to InfiniBand RDMA Reads. When an RDMA Read or Read request is issued by a transmit module, a receive module is informed of the packet sequence numbers (PSN) associated with the expected RDMA Read responses. The receive module maintains a linked list for each queue pair that issues RDMA Reads. Each entry in the linked list corresponds to one RDMA Read for the associated queue pair, and identifies the first and last PSN and includes a link to the next entry in the linked list. When the final RDMA Read response is received, the receive module notifies the transmit module, which can then retire the RDMA Read from its retry queue.