Parallel Processing Unit Interconnection Rings for Distributed Training

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methodologies for parallel/distributed training of deep neural networks face bottlenecks due to synchronization issues in minibatch stochastic gradient descent processing, particularly in inter-computing node communication, which can be alleviated by improving chip-to-chip interconnection bandwidth.

Innovation Solution

A compute system with parallel processing units organized into subsets and configured into communication rings for efficient computation of functions like Reduce and All_Reduce, utilizing high-bandwidth inter-chip networks for parallel communication, enabling hierarchical scaling and optimized computation within clusters of chips.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If synchronized minibatch SDG processing is applied on distributed computing nodes, then parallel acceleration is achieved, but communication bottlenecking occurs

Engineering Contradiction:

Improveparallel accelerationVSAvoidcommunication bottlenecking

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent segments the monolithic synchronization operation into multiple stages using ring-based all-reduce algorithms. Instead of requiring all nodes to synchronize simultaneously, the computation is divided into sequential phases where data flows through the ring topology in controlled steps, reducing peak communication bandwidth requirements while maintaining parallel processing benefits.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces ring-based communication protocols as intermediaries between computing nodes. The ring topology acts as a mediator that orchestrates data flow and synchronization, allowing nodes to communicate in a structured sequence rather than requiring direct peer-to-peer synchronization, thereby reducing communication bottlenecks.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If Reduce and All_Reduce functions are implemented using tree-based topology, then synchronization is achieved, but bandwidth utilization is suboptimal

Engineering Contradiction:

ImprovesynchronizationVSAvoidbandwidth utilization

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent inverts the conventional tree-based reduction approach by using ring-based all-reduce algorithms. Instead of aggregating data upward through a hierarchical tree structure, the patent circulates data through a ring topology where each node contributes to the reduction in a sequential manner, achieving better bandwidth utilization and scalability.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent transitions from the hierarchical two-dimensional tree structure to a cyclic one-dimensional ring structure. This dimensional change allows for more efficient use of network bandwidth by distributing communication load uniformly across all nodes in the ring, avoiding the bandwidth concentration issues inherent in tree-based approaches.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Speed

If chip-to-chip interconnection bandwidth is increased, then communication latency is reduced, but system complexity increases

Engineering Contradiction:

Improvecommunication latencyVSAvoidinterconnection complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent designs the interconnection network with universal ring-based communication protocols that can handle multiple operations (Reduce, All_Reduce, broadcast) through the same infrastructure. This multi-functionality reduces the need for specialized high-bandwidth interconnections for each operation type, thereby reducing overall system complexity while maintaining low latency performance.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20220308890A1Multi-processing unit interconnected accelerator systems and configuration techniques

Publication Date: 2022.09.29 T-HEAD (SHANGHAI) SEMICON CO LTD

AI summary

A compute system providing hierarchical scaling can include one or more sets of parallel processing units. The parallel processing units in a set can be organized into subsets of parallel processing units. Each parallel processing unit can be configurably couplable to two nearest neighbor parallel processing units in a same subset by two communication links, and each parallel processing unit can be configurably couplable to farthest neighbor parallel processing unit in the same subset by one communication link. Furthermore, each parallel processing unit can be configurably couplable to a corresponding parallel processing unit in the other subset by two communication links. The compute system can be configured by configuring the communication links of a set of parallel processing units into one or more compute clusters including a corresponding number of communication rings based on a specified compute parameter. Input data for computing on a given compute cluster divided and loaded onto respective parallel processing units of the given compute cluster. A function can be computed on the loaded input data by the given compute cluster using a parallel communication ring algorithm of the function.