Multi-Processing Unit Interconnect Topology for All_Reduce Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methodologies for parallel/distributed training of deep neural networks face bottlenecks due to synchronization issues in minibatch stochastic gradient descent processing, particularly in inter-computing node communication, which can be alleviated by improving chip-to-chip interconnection bandwidth.
Innovation Solution
A compute system with multiple parallel processing units (PPUs) is organized into subsets, each PPU connected via bi-directional communication links, enabling efficient computation of the All_Reduce function through reduced communication iterations and full utilization of communication interfaces, facilitating high-speed serial/deserialization interconnection within a cluster of chips.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If ring-based All_Reduce implementation is used with N nodes communicating 2*(N-1) times, then bandwidth utilization rate and efficiency are improved, but communication iterations and time are increased
Solution Approach 1:
The patent divides N processing units into two subsets (first subset with N/2 PPUe units and second subset with N/2 PPUo units). Each subset performs All_Reduce operations independently on different data portions, then results are combined. This segmentation reduces communication iterations from O(N) to O(log N) while maintaining high bandwidth utilization through parallel operations within each subset.
Solution Approach 2:
The patent introduces a hierarchical dimension to the communication topology by organizing PPUs into two subsets with different communication roles (PPUe for even indices, PPUo for odd indices). This dimensional organization allows simultaneous communication along multiple dimensions (within-subset and between-subset), achieving both reduced iterations and high bandwidth utilization through parallel data paths.
2Measurement precision
If frequent host-accelerator communication is used for synchronization, then processing accuracy is maintained, but communication bandwidth is consumed and bottlenecking occurs
Solution Approach 1:
The patent extracts the synchronization function from the host-accelerator communication path and implements it directly within the accelerator array using dedicated All_Reduce communication links between PPUs. This extraction eliminates the need for frequent host-accelerator synchronization communications while maintaining processing accuracy through accurate gradient aggregation within the accelerator subsystem.
Solution Approach 2:
The patent introduces intermediate aggregation points (specific PPUs acting as reducers) that collect and aggregate gradient data from other PPUs before results need to be applied. These intermediaries perform local All_Reduce operations, reducing the communication burden on the host-accelerator interface while maintaining the accuracy required for synchronized training.
3Device complexity
If straightforward tree-based implementation of Reduce and All_Reduce functions is used, then device complexity is reduced, but bandwidth utilization rate and efficiency are decreased
Solution Approach 1:
The patent implements a dynamic communication scheme where PPUe and PPUo units alternately act as data sources, aggregation targets, and communication partners based on the iteration phase. This dynamic role assignment enables the system to achieve ring-based bandwidth utilization efficiency without requiring complex static routing logic, maintaining relative implementation simplicity while dramatically improving productivity.
Data Source
AI summary
An accelerator system can include one or more clusters of eight processing units. The processing units can include seven communication ports. Each cluster of eight processing units can be organized into two subsets of four processing units. Each processing unit can be coupled to each of the other processing units in the same subset by a respective set of two bi-directional communication links. Each processing unit can also be coupled to a corresponding processing unit in the other subset by a respective single bi-directional communication link. Input data can be divided into one or more groups of four subsets of data. Each processing unit can be configured to sum corresponding subsets of the input data received on the two bi-directional communication links from the other processing units in the same subset with the input data of the respective processing unit to generate a respective set of intermediate data. Each processing unit can be configured to sum a corresponding set of intermediate data received on the one bi-directional communication link from the corresponding processing unit in the other subset with the intermediate data of the respective processing unit to generate respective sum data. Each processing unit can be configured to broadcast the sum data of the respective processing unit to the other processing units in the same subset on the respective sets of two bi-directional communication links.