RDMA Write Exchange Rendezvous Flow for NIC Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
RDMA technologies face challenges in synchronizing send and receive operations, leading to bottlenecks due to the need for out-of-band address resolution and potential retries, especially in MPI operations, which can result in inefficiencies and increased latency.
Innovation Solution
The implementation of a rendezvous flow with RDMA write exchange using work queue buffers and NICs, where a send device and receive device communicate through RDMA write operations, with a work queue buffer managing WQEs that include send and receive buffer addresses, and using fences to prevent overflow and ensure synchronization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If out-of-band address resolution is used for RDMA operations, then send and receive operations can be synchronized, but latency increases and bottlenecks occur
Solution Approach 1:
The patent extracts the address resolution process from the traditional out-of-band communication path and integrates it directly into the RDMA work queue buffer. The receive buffer address is stored directly in the WQE at the memory location specified by the send buffer address, eliminating the need for separate address resolution communications and reducing latency while maintaining synchronization.
Solution Approach 2:
The work queue buffer acts as an intermediary structure that directly links send and receive operations. By storing the receive buffer address within the WQE in the work queue buffer, the system creates a direct mapping between send and receive operations without requiring external communication, thus reducing latency while ensuring proper synchronization.
2Productivity
If RDMA write operations are used for data transfer, then throughput increases, but synchronization challenges arise leading to retries
Solution Approach 1:
The patent applies preliminary action by pre-storing the receive buffer address in the work queue buffer before the actual data transfer occurs. The WQE is prepared with all necessary address information in advance, allowing the RDMA write operation to proceed directly without synchronization challenges or retries, thus maintaining both high throughput and reliability.
3Productivity
If work queue buffers are used to manage RDMA operations, then operation efficiency improves, but complexity of managing work queue occupancy increases
Solution Approach 1:
The patent merges the address resolution function with the work queue buffer structure. By combining the send buffer address, receive buffer address, and work queue management into a single integrated WQE structure, the system simplifies occupancy management while maintaining high operation efficiency. The fence mechanism is integrated directly into the buffer structure, reducing the complexity of separate management systems.
Data Source
AI summary
A system includes a first processing device and a second processing device, each of which is coupled to a NIC implemented with an RDMA interface. The NICs are capable of rendezvous flows of RDMA write exchange. In an example where the first NIC is at the sender side and the second NIC is at the receiver side, a rendezvous flow is initiated by an execution of a RDMA write operation by the second NIC. The second NIC provides at least an address of a buffer in the second processing device to the first NIC through the RDMA write operation. Then the first NIC initiates a RDMA write operation to send data in a buffer in the first processing device to the second NIC. The second NIC may acknowledge receipt of the data with the second NIC. The second NIC can update a CI of the WQE based on the acknowledgement.