NIC Pipeline Aggregation Reducing Allreduce Delay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed deep learning, the communication overhead in Allreduce operations leads to unnecessary processing delays due to data transfer not directly related to aggregation calculation processing, particularly in heterogeneous computing systems where GPUs and CPUs are interconnected.
Innovation Solution
A distributed processing system with a pipeline connection configuration that performs aggregation calculation and distribution processing using a Ring-Allreduce algorithm, where each communication interface handles data in a specific order and starts distribution only when aggregation is complete, reducing the number of communication ports required and optimizing data transfer paths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is transferred from tail NIC to head NIC before distribution in conventional Ring-Allreduce, then aggregation calculation can be performed, but unnecessary processing delay occurs due to data transfer not directly related to aggregation
Solution Approach 1:
The patent applies preliminary action by having each NIC perform aggregation calculation on received data before the complete aggregation result is available from all processors. This allows the distribution phase to begin earlier, overlapping aggregation and distribution operations to reduce total processing delay.
Solution Approach 2:
The patent inverts the conventional sequence by starting distribution from the tail NIC rather than waiting for head NIC to receive and forward data. This reversal eliminates unnecessary data transfer delays and allows parallel execution of aggregation and distribution phases.
2Productivity
If aggregation calculation processing and data distribution processing are pipelined, then processing delays caused by communication overhead can be reduced, but the system requires optimized coordination between multiple NICs
Solution Approach 1:
The patent segments the Allreduce operation into distinct aggregation and distribution phases, with each NIC independently handling both phases. This segmentation enables pipelining where distribution can begin before aggregation completes for all processors, improving throughput while maintaining manageable coordination through clear phase separation.
3Speed
If RDMA is used for direct data storage from network to GPU memory, then data movement delay is extremely low, but communication overhead occurs when data movement cooperates with processing
Solution Approach 1:
The patent merges data movement and aggregation calculation operations by having each NIC perform aggregation on received data while simultaneously preparing for distribution. This combining of operations eliminates idle communication overhead delays while utilizing the high-speed RDMA data movement capability, as processing begins immediately upon data receipt without waiting for separate communication cycles.
Data Source
AI summary
Each NIC performs an aggregation calculation of data output from each processor in a normal order including a head NIC located at a head position of a first pipeline connection, an intermediate NIC located at an intermediate position, and a tail NIC located at a tail position, and when the aggregation calculation in the tail NIC is completed, each NIC starts distribution of an obtained aggregation result, distributes the aggregation result in a reverse order including the tail NIC, the intermediate NIC, and the head NIC, and outputs the aggregation result to the processor of the communication interface.