Full Mesh Network Topology for All-Reduce Communication Contention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel computer systems face route contention issues during all-reduce communication, which leads to increased communication time, especially in network topologies like general tree structures, and cannot effectively avoid contention in all communication types beyond all-to-all communications.
Innovation Solution
A single-layer full mesh system with a regular and fixed routing infini-band network is implemented, where each switch is connected to other switches in a full mesh topology, and each server is allocated a unique number for communication, using exclusive OR operations to generate identifiers for communication phases, ensuring no packets are transmitted in the same direction simultaneously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a general tree structure or fat tree structure is used for network topology, then all-to-all communication can be performed, but route contention occurs during all-reduce communication
Solution Approach 1:
The patent changes the network topology parameter from tree structure to full mesh structure, where each switch is directly connected to every other switch. This structural parameter change eliminates route contention during all-reduce communication by providing multiple independent paths between any two nodes, allowing simultaneous packet transmission without interference.
2Productivity
If full mesh system is used to avoid route contention in all-to-all communication, then communication efficiency is improved, but the technique cannot avoid route contention in other communication types like all-reduce
Solution Approach 1:
The full mesh topology provides universal connectivity that serves multiple communication patterns effectively. The same topology structure that optimizes all-to-all communication also naturally handles all-reduce, reduce-scatter, and other collective operations without route contention, making the system versatile across different communication types.
3Reliability
If more switches are added to avoid route contention, then communication reliability is improved, but construction cost increases
Solution Approach 1:
The patent segments the communication operations into different phases (all-to-all phase and all-reduce phase) and applies the full mesh topology specifically optimized for each phase. This segmentation allows the system to achieve route contention avoidance without requiring an excessively large number of switches, as the topology is tailored to the specific communication patterns of each phase.
Data Source
AI summary
An information processing system includes a plurality of switches connected to each other in a form of a full mesh and a plurality of information processing apparatuses respectively connected to any one of the plurality of switches. A first information processing apparatus of the plurality of information processing apparatuses includes a processor. The processor is configured to generate a second identifier by calculating an exclusive OR of a first identifier and a first number corresponding to a communication phase. The first identifier identifies a first switch connected to the first information processing apparatus. The first number is included in a set of linearly independent numbers allocated to the first information processing apparatus. The processor is configured to perform communication with a second information processing apparatus of the plurality of information processing apparatuses. The second information processing apparatus is connected to a second switch having the second identifier.