Interleaved Compute Communication for Distributed Neural Network Training

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current parallel graphics data processing systems face challenges in efficiently training neural networks due to limitations in processing power and data distribution across multiple interconnected worker nodes.

Innovation Solution

The system employs a distributed training network with multiple interconnected worker nodes, utilizing graphics processing units (GPUs) to accelerate neural network training through parallel processing techniques such as SIMT architectures and high-speed interconnects like PCIe or NVLink.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If distributed training network with multiple worker nodes is used, then training speed and efficiency are improved, but system complexity increases

Engineering Contradiction:

Improvetraining speedVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the neural network training workload across multiple worker nodes, with each node handling specific layers or batches. This segmentation enables parallel processing that improves training speed while managing system complexity through modular architecture.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a new dimension of parallelism by utilizing multiple interconnected worker nodes in a distributed network, transitioning from single-node sequential processing to multi-node parallel processing, thereby significantly enhancing training efficiency.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Power

If parallel processing techniques are employed, then processing power is enhanced, but coordination overhead increases

Engineering Contradiction:

Improveprocessing powerVSAvoidcoordination overhead

Core Design Contradiction:

PowerVSLoss of time

Solution Approach 1:

The patent performs preliminary actions by pre-configuring the distributed worker nodes and establishing communication protocols before training begins. This preparation reduces coordination overhead during actual parallel processing operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent maintains continuous useful action across all worker nodes through efficient load balancing and data distribution mechanisms, ensuring that parallel processing power is maximized while minimizing idle time and coordination delays.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentEP4089537B1Fine-grain compute communication execution for deep learning frameworks

Publication Date: 2025.03.12 INTEL CORP

Figure 1
Figure 2A
Figure 2B

AI summary

One embodiment provides for a system to configure distributed training of a neural network. The system includes memory to store a library to facilitate transmission of data during distributed training of the neural network; a network interface to transmit and receive gradient data associated with the trainable parameters; a general-purpose processor to execute instructions provided by the library, the instructions to cause the general-purpose processor to configure the network interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework; and a graphics processor to perform compute operations associated with machine learning framework workflow to generate the gradient data associated with the trainable parameters, wherein, based on the machine learning framework workflow, the library is to interleave the compute operations on the graphics processor with transmission and receipt of gradient data via the network interface.