Directed Spanning Trees for GPU Parameter Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data-parallel deep neural network (DNN) training faces significant communication bottlenecks due to model parameter synchronization across GPUs, exacerbated by increasing model sizes and faster GPU computation, leading to prolonged training times.
Innovation Solution
The implementation of dynamically generated communication primitives using packed directed spanning trees to optimize parameter synchronization, leveraging heterogeneous inter-GPU communication channels for hybrid and faster data transfers, and determining an optimal chunk size for data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data-parallel training is used with multiple GPUs, then training speed is improved, but communication overhead increases significantly
Solution Approach 1:
The communication graph is segmented into multiple directed spanning trees, each responsible for synchronizing parameters between specific GPU subsets. This segmentation allows parallel parameter synchronization operations to proceed simultaneously across different tree structures, reducing total communication time and overhead during training.
Solution Approach 2:
The patent transitions from traditional undirected communication graphs to directed spanning trees, adding a directional dimension to parameter synchronization. This enables optimized data flow paths where parameters are pushed unidirectionally through tree edges, improving communication efficiency and reducing redundant data transfers across the GPU cluster.
2Measurement precision
If model size increases to improve accuracy, then accuracy is improved, but training time increases
Solution Approach 1:
The system performs preliminary actions by pre-computing and packing multiple directed spanning trees before parameter synchronization begins. This preliminary graph generation and optimization allows the actual parameter synchronization to proceed more efficiently without computational delays, enabling faster training of larger models.
Solution Approach 2:
The patent ensures continuous parameter synchronization by maintaining multiple active directed spanning trees that operate simultaneously. Rather than sequential updates, the system continues useful action through parallel tree-based synchronization, keeping all GPU nodes continuously updated without idle waiting periods, thus reducing total training time for large models.
3Power
If GPU computation speed increases, then computational performance is improved, but communication bottleneck becomes more severe
Solution Approach 1:
The system changes communication parameters by switching from traditional broadcast protocols to directed spanning tree-based parameter pushing. This parameter change in the communication protocol allows the system to scale better with computational speed, as the directed tree structure minimizes data transfer distance and reduces synchronization latency, keeping communication speed adequate even as GPU computational performance increases.
4Speed
If more communication links are used for parameter synchronization, then synchronization speed is improved, but system complexity increases
Solution Approach 1:
The patent merges multiple communication paths into a unified directed spanning tree structure. Instead of managing separate complex communication topologies, the system combines parameter synchronization across all GPUs into a hierarchical tree-based network, reducing topological complexity while maintaining high synchronization speed through the structured merge of communication paths.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
Technologies are disclosed herein for dynamically generating communication primitives for use in model parameter synchronization during data-parallel DNN training by packing directed spanning trees. An interconnect topology for communication between GPUs in a computing system is determined. A quantity of directed spanning trees are generated for transmitting data between the GPUs using the interconnect topology and packed. The directed spanning trees define the connections between GPUs that are to be utilized for the transmission and the amount of data to be transmitted on each connection. Program code is generated for implementing the data transfer defined by the directed spanning trees. When the program code is executed, the directed spanning trees are used to pipeline the transmission of chunks of data, such as model parameters used during data-parallel DNN training, between the GPUs. The program code can also determine an optimal chunk size for data to be transferred between the GPUs.