Hyper-Square Interconnect Topology for Ring-AllReduce in Distributed Deep Learning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional interconnect topologies like the torus topology face issues with scalability and communication delays due to long wiring, which hinder efficient data synchronization and AllReduce operations in distributed neural network training.

Innovation Solution

The hyper-square interconnect topology and advanced ring-based AllReduce operations are introduced, utilizing ring connections formed through inter-group and intra-group interconnects to synchronize data across computing nodes, eliminating the need for long wires and enabling efficient data transfer.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Shape

If torus topology is used for interconnect, then computing nodes can be connected in a structured manner, but communication delays increase due to long wirings

Engineering Contradiction:

Improveinterconnect topology structureVSAvoidcommunication delay

Core Design Contradiction:

ShapeVSLoss of time

Solution Approach 1:

The system segments computing nodes into multiple groups, with each group forming an independent ring structure. This segmentation allows data to travel through shorter paths within each ring, reducing communication delays compared to a single large torus structure where data must traverse longer wiring paths.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a hierarchical dimension by organizing rings into groups and establishing inter-group connections. This creates a multi-dimensional communication structure where data can switch between intra-group ring communication and inter-group shortcut connections, effectively reducing the physical wiring length and communication delay.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Shape

If torus topology is used for interconnect, then computing nodes are connected in a fixed structure, but scalability is limited due to inability to divide computing nodes for multiple tasks

Engineering Contradiction:

Improveinterconnect topology structureVSAvoidscalability and task assignment flexibility

Core Design Contradiction:

ShapeVSAdaptability or versatility

Solution Approach 1:

Computing nodes are segmented into multiple independent groups, each capable of forming its own ring structure. This segmentation enables different groups to be assigned to different computing tasks simultaneously, improving scalability and resource utilization. The system can dynamically allocate groups to various tasks based on workload requirements.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system implements dynamic task assignment by allowing computing nodes to be divided into different groups based on task requirements. Groups can be reconfigured, expanded, or contracted dynamically, enabling the interconnect structure to adapt to varying computational workloads and improve overall system versatility.

Inventive Principle:

Principle #15Dynamics

3Reliability

If conventional AllReduce algorithm is used, then data synchronization is achieved, but bandwidth utilization is inefficient

Engineering Contradiction:

Improvedata synchronizationVSAvoidbandwidth utilization efficiency

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The AllReduce operation is segmented into independent ring-based operations within each group. This allows parallel execution of reduction operations across multiple rings simultaneously, improving bandwidth utilization. Each ring can independently perform data synchronization without contending for the same communication resources, thereby increasing overall system efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The ring-based AllReduce algorithm enables continuous data flow around the ring structure, allowing multiple data elements to be reduced in parallel as they circulate. This continuous action maximizes bandwidth utilization by keeping communication channels actively engaged in useful reduction operations throughout the synchronization process.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS12124889B2Efficient and more advanced implementation of ring-allreduce algorithm for distributed parallel deep learning

Publication Date: 2024.10.22 T-HEAD (SHANGHAI) SEMICON CO LTD

AI summary

The present disclosure provides a method for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group of the plurality of groups with a computing node A of a second group neighboring the first group, a computing node B of the first group with a computing node B of the second group, a computing node C of the first group with the computing node C of the second group, and a computing node D of the first group with a computing node D of the second group, the method comprising: syncing across a first dimension of computing nodes using a first set of ring connections, wherein the first set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes along the first dimension; and broadcasting synced data across a second dimension of computing nodes using a second ring connection.