Parallel GPU Training with Preliminary Parameter Exchange

Solution Overview

Problem

In deep learning training processes using multiple GPUs, the exchange of modification parameters between GPUs is time-consuming, leading to increased waiting times and reduced acceleration ratios due to the sequential nature of parameter updates.

Innovation Solution

A method and apparatus that parallelize the training and parameter exchange processes by loading training data sets to multiple GPUs and controlling them to generate and exchange modification parameters concurrently, ensuring that exchange processing is completed before the end of each training round.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple GPUs are used to perform deep learning training in parallel, then training speed is improved, but parameter exchange time increases waiting time and reduces acceleration ratio

Engineering Contradiction:

Improvetraining speedVSAvoidparameter exchange waiting time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs parameter exchange operations in advance during the training process, specifically exchanging parameters from earlier layers before the training round completes. This preliminary action ensures that critical parameters are available sooner, reducing the overall waiting time for parameter synchronization while maintaining the parallel training benefit.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts the parameter exchange strategy by selectively exchanging parameters from different layers at different times during training. Rather than uniform exchange, it prioritizes exchanges based on layer importance and training progress, optimizing the balance between communication overhead and training efficiency.

Inventive Principle:

Principle #15Dynamics

2Reliability

If parameter exchange is performed sequentially after training, then accuracy is maintained, but overall training time increases

Engineering Contradiction:

Improvemodel accuracyVSAvoidtraining duration

Core Design Contradiction:

ReliabilityVSDuration of action of moving object

Solution Approach 1:

The system performs parameter exchange operations in advance during the training process, specifically exchanging parameters from earlier layers before the training round completes. This preliminary action ensures that critical parameters are available sooner, reducing the overall waiting time for parameter synchronization while maintaining the parallel training benefit.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system maintains continuous training operations across multiple GPUs by overlapping parameter exchange with ongoing training computations. Instead of stopping training to exchange parameters, the system keeps the training process continuous and uninterrupted, maximizing resource utilization and reducing idle time.

Inventive Principle:

Principle #20Continuity of useful action

3Stability of the object's composition

If all modification parameters are exchanged after training completion, then synchronization is achieved, but waiting time is maximized

Engineering Contradiction:

Improveparameter synchronizationVSAvoidwaiting time

Core Design Contradiction:

Stability of the object's compositionVSLoss of time

Solution Approach 1:

The system performs parameter exchange operations in advance during the training process, specifically exchanging parameters from earlier layers before the training round completes. This preliminary action ensures that critical parameters are available sooner, reducing the overall waiting time for parameter synchronization while maintaining the parallel training benefit.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system exchanges only the necessary parameters from specific layers during training rather than waiting to exchange all parameters after training completes. This partial action approach exchanges parameters from earlier layers selectively, achieving sufficient synchronization without the full overhead of complete parameter exchange, thus reducing waiting time while maintaining adequate parameter consistency.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS10943324B2Data processing method, apparatus, and electronic device

Publication Date: 2021.03.09 TENCENT TECHNOLOGY (SHENZHEN) CO LTD

AI summary

A data processing method is described. A processor provides different training data sets to a plurality of graphics processing units (GPUs), respectively. The processor controls the plurality of GPUs to generate respective sets of modification parameters by performing respective training processes in a parallel manner using respectively managed versions of a predictive model according to the corresponding training data sets. The processor controls the plurality of GPUs to exchange, before completion of the respective training processes, at least a portion of the sets of modification parameters that have been generated by the plurality of GPUs. The processor also causes the plurality of GPUs to modify the respectively managed versions of the predictive model according to at least the portion of the sets of modification parameters exchanged among the plurality of GPUs.