Receiver-Based RDMA Scheduling for AI/ML Incast Congestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Incast congestion occurs in high-performance AI/ML networks due to multiple synchronized accelerators sending data to the same receiver simultaneously, leading to buffer buildup, congestion, and increased tail latency, which current technologies fail to effectively manage.
Innovation Solution
Implementing receiver-based traffic scheduling by accessing receive work queue entities (RWQEs) associated with send work queue entities (SWQEs), determining a transmission rate based on available bandwidth and transmitter count, and transmitting rate control notifications (RCNs) to adjust transmission rates dynamically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple synchronized accelerators send data to the same receiver in parallel, then data transfer throughput increases, but buffer buildup and congestion occur
Solution Approach 1:
The receiver device performs preliminary actions by determining the transmission rate before data transfer begins. The system calculates the optimal transmission rate based on available bandwidth and transmitter count, then communicates this rate to transmitters in advance, preventing buffer buildup before it occurs
Solution Approach 2:
The system implements feedback by continuously monitoring data transfer conditions and adjusting transmission rates dynamically. The receiver device determines transmission rates based on real-time bandwidth availability and transmitter count, then communicates these rates back to transmitters to maintain optimal flow and prevent congestion
2Loss of time
If transmission rate is increased to reduce latency, then data transfer speed improves, but buffer buildup increases
Solution Approach 1:
The system dynamically changes transmission rate parameters based on real-time network conditions. The receiver device calculates optimal transmission rates by considering available bandwidth and transmitter count, adjusting the rate parameter to balance transfer speed with buffer utilization, preventing both excessive latency and buffer overflow
3Loss of time
If receiver-based scheduling is implemented to manage congestion, then tail latency decreases, but system complexity increases
Solution Approach 1:
The receiver device performs self-service by autonomously determining transmission rates based on its own bandwidth availability and the number of active transmitters. This self-service approach eliminates the need for complex centralized control or sophisticated protocols, reducing overall system complexity while effectively managing tail latency
Solution Approach 2:
The receiver device performs multiple functions using a single mechanism: it monitors bandwidth availability, counts transmitter devices, calculates optimal transmission rates, and communicates these rates to transmitters. This multi-functional approach avoids the need for separate specialized components, reducing system complexity
Data Source
AI summary
A system for Remote Direct Memory Access incast congestion management is provided. The system may access one or more receive work queue entities (RWQEs) associated with one or more send work queue entities (SWQEs), determine a transmission rate based on an available bandwidth and a number of transmitter devices associated with the one or more SWQEs, and transmit a rate control notification (RCN) message to a transmitter device associated with a send work queue entity of the one or more SWQEs, wherein the rate control notification message comprises the determined transmission rate.