Global Top-k Sparsification for Low Bandwidth Distributed Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for distributed training of deep neural networks face inefficiencies in data communication due to high communication complexity when using Top-k sparsification on low-bandwidth networks, particularly with large numbers of workers, as they require transferring irregularly indexed sparse gradients, leading to significant communication overhead.
Innovation Solution
The implementation of a global Top-k (gTop-k) sparsification mechanism that uses a tree structure to select and aggregate the global Top-k gradients, reducing communication complexity from O(Pmρ) to O(log2(P)mρ, and integrating this approach into the gTopKAllReduce algorithm for efficient gradient aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If Top-k sparsification is used to reduce gradient volume, then communication bandwidth requirement is reduced, but communication complexity increases due to irregular indices making aggregation difficult
Solution Approach 1:
The patent segments the gradient aggregation process into multiple communication rounds, where in each round workers communicate with specific peers identified by peerMasks. This divides the complex task of aggregating sparse gradients with irregular indices into manageable steps, reducing the complexity of each individual communication operation.
Solution Approach 2:
The patent introduces peerMasks as an intermediary data structure that mediates between workers with different sparse gradient indices. The peerMasks enable workers to identify appropriate communication partners and manage the aggregation process, simplifying the complexity of handling irregular indices without requiring full AllGather operations.
2Productivity
If AllGather is used to accumulate sparse gradients, then gradient aggregation is achieved, but communication complexity becomes O(kP) which is inefficient on low bandwidth networks
Solution Approach 1:
The patent makes the communication pattern dynamic by adjusting peerMasks across different communication rounds. In each round, workers dynamically identify their communication partners based on the current peerMasks, enabling efficient gradient aggregation without fixed communication patterns. This dynamic approach optimizes the aggregation process for sparse gradients with irregular indices.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and maintaining peerMasks that indicate communication partners for each worker. This preliminary preparation of communication routing information eliminates the need for complex on-the-fly peer selection during gradient aggregation, reducing communication time on low bandwidth networks.
3Productivity
If data parallelism is used to accelerate training, then computational efficiency is improved, but communication overhead increases due to gradient exchange in each iteration
Solution Approach 1:
The patent changes the parameter of gradient representation from dense to sparse format, retaining only the top-k non-zero gradient elements along with their indices. This parameter change reduces the volume of data that needs to be communicated during each iteration, thereby reducing communication energy consumption while maintaining the speedup benefits of data parallelism.
Data Source
AI summary
The present invention relates to a system for efficient large-scale data distribution in a distributed and parallel processing environment. In particular, the present invention relates to global Top-k sparsification for low bandwidth networks. The present invention verifies that gTop-k S-SGD has nearly consistent convergence performance with S-SGD and evaluates the training efficiency of gTop-k on a cluster with 32 GPU machines which are inter-connected with 1 Gbps Ethernet. The experimental results show that the present invention achieves up to 2.7-12× higher scaling efficiency than S-SGD with dense gradients, and 1.1-1.7× improvement than the existing Top-k S-SGD.