Processing Node Clique Ring Topology for Allreduce Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning and artificial intelligence applications face inefficiencies in data exchange between processing nodes, particularly in implementing the Allreduce collective operation, which is crucial for converging model parameters across distributed processing units during neural network training.
Innovation Solution
A computer architecture with processing nodes arranged in cliques and interconnected in rings, allowing for efficient data exchange through a combination of reduce-scatter and Allgather phases, optimizing the implementation of the Allreduce collective by ensuring concurrent data transmission and reduction across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data exchange is performed sequentially between processing nodes using traditional Allreduce implementation, then implementation simplicity is maintained, but data exchange latency increases and productivity decreases
Solution Approach 1:
The patent segments the Allreduce operation into two distinct phases: reduce-scatter phase where data is reduced and distributed to different nodes, and Allgather phase where data is collected from all nodes. This segmentation allows concurrent execution of multiple data exchange operations across different node pairs, reducing overall latency and improving productivity.
Solution Approach 2:
The patent introduces a dimensional change by organizing processing nodes into a multi-dimensional topology (e.g., 2D grid, 3D cube) rather than a simple linear sequence. This allows data exchange to occur simultaneously along multiple dimensions, enabling parallel reduction and gathering operations across different node pairs, thereby reducing latency and improving throughput.
2Productivity
If more processing nodes are connected in sequence to increase bandwidth utilization, then data exchange capacity improves, but system complexity increases and power consumption rises
Solution Approach 1:
The patent employs dynamic routing and adaptive phase switching in the data exchange process. During the reduce-scatter phase, data flows dynamically through the network topology to reach destination nodes, and the system adapts the execution flow between phases based on data availability and node readiness. This dynamic approach maximizes bandwidth utilization without requiring a overly complex static topology.
3Loss of energy
If traditional Allreduce is implemented with sequential reduce then broadcast, then implementation simplicity is maintained, but energy efficiency deteriorates due to redundant data transmission
Solution Approach 1:
The patent extracts the broadcast operation from the traditional Allreduce sequence and separates it into the Allgather phase. Instead of performing a full broadcast after reduction, the system only gathers the necessary reduced data from nodes that have computed it. This extraction eliminates redundant data transmission where full datasets are broadcast to all nodes, reducing energy consumption while maintaining data exchange efficiency.
Data Source
AI summary
A computer comprising a plurality of processing nodes is provided. Each processing node has at least one processor configured to process input data to generate an array of data items. The processing nodes are arranged in cliques in which each processing node of a clique is connected to each other processing node in the clique by first and second clique links. The cliques are inter-connected in rings such that each processing node is a member of a single clique and a single ring. The processing nodes of all cliques are configured to exchange in each exchange step of a machine learning collective via the respective first and second clique links at least two data items with the other processing node(s) in its clique, and all processing nodes are configured to reduce each received data item with the data item in the corresponding position in the array on that processing node.