Computing Cluster Protocol Switching Under High RDMA Packet Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently utilizing multiple distributed computing power clusters for large-scale model training is exacerbated by significant cross-machine-room communication time, necessitating a solution to minimize its impact on training performance.
Innovation Solution
A communication method and apparatus that dynamically switches between Remote Direct Memory Access (RDMA) and Transmission Control Protocol (TCP) based on packet loss rates to optimize data transmission efficiency across and within computing power clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If multiple distributed computing power clusters are used for large-scale model training, then computing power and model training capability are improved, but cross-machine-room communication time increases significantly
Solution Approach 1:
The system dynamically switches between RDMA and TCP communication protocols based on real-time packet loss rate detection. When packet loss exceeds a threshold, the system transitions from RDMA to TCP to maintain communication reliability, and vice versa. This dynamic adaptation optimizes the balance between communication speed and reliability across distributed clusters.
Solution Approach 2:
The invention changes the communication protocol parameter (from RDMA to TCP or vice versa) based on network conditions indicated by packet loss rate. This parameter change allows the system to adapt to varying network environments in distributed clusters, minimizing communication time impact while maintaining reliability.
2Speed
If RDMA protocol is used for communication, then data transmission speed is improved, but reliability deteriorates when packet loss rate is high
Solution Approach 1:
The system continuously monitors packet loss rate and uses this feedback to determine whether to maintain RDMA protocol or switch to TCP protocol. When packet loss rate exceeds the threshold, the feedback mechanism triggers a protocol switch to TCP, which provides better reliability through acknowledgment and retransmission mechanisms.
Solution Approach 2:
The packet loss rate threshold acts as an intermediary condition that mediates between RDMA and TCP protocols. Based on whether the packet loss rate exceeds this threshold, the system selects the appropriate protocol, allowing RDMA for high speed when conditions are good and TCP for reliability when conditions deteriorate.
3Reliability
If TCP protocol is used for communication, then reliability is improved, but data transmission speed decreases
Solution Approach 1:
The system dynamically selects between TCP and RDMA protocols based on real-time network conditions. When packet loss rate is below the threshold, the system switches to RDMA protocol to maximize data transmission speed. This dynamic selection ensures TCP reliability when needed while capturing RDMA speed benefits when network conditions permit.
Data Source
AI summary
A communication method, apparatus, electronic device and storage medium for a computing power cluster are provided. An implementation of the method includes: during a process of communicating with a communication receiver using Remote Direct Memory Access (RDMA) protocol, obtaining a first packet loss rate corresponding to the RDMA protocol; in response to the first packet loss rate being higher than a first preset packet loss rate, initiating a first handshake request to the communication receiver for requesting to switch to Transmission Control Protocol (TCP) for communication; receiving a first handshake response returned by the communication receiver for the first handshake request, and determining a first starting transmission position of data according to a last data receiving position in the first handshake response; and communicating, by using the TCP, with the communication receiver starting from data corresponding to the first starting transmission position.