RDMA Parameter Tuning for Cross-Cluster Bandwidth Saturation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of accommodating large computing power clusters with hundreds of thousands of cards in a single machine room is impractical, and utilizing multiple clusters with thousands of cards can reduce construction costs and efficiently utilize fragmented clusters, but requires effective cross-computing power cluster communication solutions.
Innovation Solution
A method for cross-computing power cluster communication involving increasing RDMA connection count and Buffer size until actual bandwidth no longer increases, determining transmission parameters, and communicating across different clusters using these parameters to enhance efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If multiple clusters with thousands of cards are utilized to reduce construction costs and efficiently utilize fragmented clusters, then construction costs are reduced and computing power supply is activated, but cross-cluster communication efficiency deteriorates
Solution Approach 1:
The patent dynamically adjusts RDMA connection counts and Buffer sizes based on detected actual bandwidth values. By changing these communication parameters adaptively, the system optimizes cross-cluster communication efficiency while maintaining the cost benefits of using multiple fragmented clusters.
Solution Approach 2:
The system implements feedback mechanisms by detecting actual bandwidth values during communication and using this information to adjust RDMA connection counts and Buffer sizes. This closed-loop control ensures communication efficiency is maintained while utilizing multiple clusters.
2Productivity
If RDMA connection count and Buffer size are increased to improve cross-cluster communication efficiency, then data transmission volume increases, but system complexity and resource consumption increase
Solution Approach 1:
The patent employs dynamic adjustment of RDMA connection counts and Buffer sizes based on real-time bandwidth detection. Rather than using fixed high values that would increase complexity, the system adapts parameters to actual communication needs, maintaining data transmission volume while reducing unnecessary system complexity.
Solution Approach 2:
The system changes communication parameters (RDMA connection count, Buffer size) dynamically based on detected bandwidth conditions. This allows the system to achieve high data transmission volume only when necessary, avoiding the constant high complexity that would result from permanently elevated parameter settings.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
A method and apparatus for cross-computing power cluster communication, an electronic device, and a computer readable storage medium, are provided. An implementation of the method includes: in response to a communication initiator and a communication receiver respectively belonging to different computing power clusters, increasing a RDMA connection count on the basis of an initial connection count until a detected actual bandwidth value no longer increases, to obtain a target connection count; increasing, with the RDMA connection count being maintained at the target connection count, a Buffer size on the basis of an initial size until a detected actual bandwidth value no longer increases, to obtain a target size; determining cross-cluster transmission parameters, based on the target connection count and the target size; and communicating, cross-cluster transmission parameters, with the communication receiver belonging to a different computing power cluster.