Latin Square Fat-Tree Network Topology for Allreduce Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel computers, the efficiency of communication is hindered by route competition in Allreduce communication, which occurs when multiple packets are transmitted simultaneously in the same direction, leading to increased communication time, especially when using traditional tree structures that require a larger number of switches.
Innovation Solution
Implementing a Latin square fat-tree network topology with regular and fixed routing in InfiniBand networks, where switches are coupled in a Latin square fat-tree structure, and using communication tables to manage packet transmission, ensuring that packets are routed without simultaneous transmission in the same direction, thereby avoiding route competition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional tree structure is used for Allreduce communication, then communication coverage can be achieved, but route competition occurs causing increased communication time
Solution Approach 1:
The patent segments the Allreduce communication process into multiple phases: reduce phase (where partial results are computed and aggregated) and scatter phase (where final results are distributed). This segmentation allows different groups of servers to perform operations simultaneously without conflicting over the same communication routes, thereby eliminating route competition while maintaining comprehensive communication coverage.
2Productivity
If greater number of servers are coupled to improve parallel processing, then processing capacity increases, but number of switches required increases
Solution Approach 1:
The patent employs a two-stage fat-tree topology where switches serve multiple functions: Leaf switches connect servers and perform initial reduce operations, while Spine switches aggregate results and enable communication between different Leaf switch groups. This multi-functional design allows the same switch infrastructure to handle both compute coordination and data aggregation, supporting expansion to more servers without linearly increasing switch count.
3Device complexity
If Latin square fat-tree topology is used to reduce switch count, then construction cost decreases, but route competition in Allreduce communication increases
Solution Approach 1:
The patent introduces dynamic phase-based communication control in the Latin square fat-tree topology. During the reduce phase, only specific server groups communicate through designated routes, and during the scatter phase, communication patterns change to distribute results. This dynamic temporal separation of communication patterns eliminates simultaneous route conflicts while preserving the cost-efficient Latin square fat-tree structure.
Data Source
AI summary
An information processing system includes switches coupled to each other in a form of a Latin square fat-tree and apparatuses coupled to the switches. A processor of a first apparatus coupled to one of first switches executes a first reduce with others of the first apparatuses. A processor of a second apparatus coupled to a representative switch executes a second reduce with representative apparatuses of respective switches belonging to a group corresponding to the second apparatus. The processor of the second apparatus executes Allreduce with others of the second apparatuses and transmits the result of the Allreduce to the representative apparatuses of the respective switches belonging to the group corresponding to the second apparatus. A processor of a representative apparatus that receives the result of the Allreduce transmits the result of the Allreduce to others of apparatuses coupled to a switch to which the representative apparatus is coupled.