Distributed Deep Learning With Multi-NIC Allreduce Communication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The bottleneck in Allreduce processing due to insufficient network interface cards (NICs) for accelerators in distributed deep learning systems causes congestion and delays, hindering efficient distributed deep learning performance.

Innovation Solution

A computer system with multiple accelerators and network interface circuits that enable parallel transmission and reception of learning results, allowing for parallel communication and reduced overhead between computers, thereby optimizing distributed deep learning.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple accelerators are used for distributed deep learning, then processing capability is improved, but communication bottleneck occurs due to insufficient NICs

Engineering Contradiction:

Improveprocessing capabilityVSAvoidcommunication delay

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system segments the communication task by assigning each NIC to handle communication for a specific subset of accelerators. This segmentation allows parallel communication channels to operate simultaneously, eliminating the bottleneck caused by insufficient NICs while maintaining high processing capability across multiple accelerators.

Inventive Principle:

Principle #1Segmentation

2Reliability

If Allreduce processing is performed for weight data sharing, then model convergence is improved, but communication overhead increases

Engineering Contradiction:

Improvemodel convergenceVSAvoidcommunication overhead

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The Allreduce processing is segmented across multiple NICs, with each NIC handling the communication task for its assigned accelerators independently. This segmentation distributes the communication overhead across parallel channels, reducing the total communication overhead while maintaining the model convergence benefits of weight data sharing.

Inventive Principle:

Principle #1Segmentation

3Quantity of substance

If data is distributed among multiple computers, then storage bottleneck is relieved, but coordination complexity increases

Engineering Contradiction:

Improvedata storage capacityVSAvoidcoordination complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent introduces a coordination mechanism where the CPU acts as an intermediary to manage the distributed data processing across multiple computers. The CPU coordinates the parallel processing tasks and manages data distribution, simplifying the coordination complexity while enabling the system to utilize the expanded storage capacity of multiple computers.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12450481B2Distributed processing computer and distributed deep learning system

Publication Date: 2025.10.21 NT T INC

AI summary

An embodiment is a computer including a plurality of accelerators, a computer for distributed processing includes a plurality of accelerators to each of which a part of a neural network is assigned and each of which is configured to derive a learning result based on input data and update each parameter value included in the part of the neural network by using the learning result; a plurality of network interface circuits each of which is configured to transmit and receive information on learning including the learning result via a network, and an arithmetic processing unit that is configured to control the plurality of accelerators and the plurality of network interface circuits to cause each of the plurality of accelerators to derive a learning result based on input data and to cause the plurality of network interface circuits to transmit and receive information on learning including the learning result.