In-Network Wave Synchronization for Scalable Collective Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current collective communication primitives in distributed machine learning/deep learning training algorithms face scalability issues due to high latency and synchronization overhead, particularly in large-scale systems, where the ring-scheme algorithm's latency is proportional to the number of GPUs and synchronization mechanisms become bottlenecks.
Innovation Solution
Implementing a credit-based policy to offload collective communication primitives into network elements such as switches or routers, allowing for the injection of collective communication operations like reduction operations directly within the network, rather than at endpoints, using a wave controller and offload engine to manage credits and operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the ring-scheme algorithm is used for collective communication, then the bandwidth is sufficient, but the latency becomes proportional to the number of GPUs and synchronization overhead increases
Solution Approach 1:
The patent segments the collective communication operations into independent waves that can be processed in parallel by network elements. Instead of sequential ring-based communication, multiple communication waves are injected simultaneously into the network, each wave handling a portion of the reduction operation independently, thereby reducing overall latency while maintaining bandwidth utilization.
Solution Approach 2:
The patent introduces network elements (switches, routers) as intermediaries that can perform reduction operations directly within the network fabric. These intermediary devices receive data from multiple GPUs, perform local reduction computations, and forward results, eliminating the need for synchronized ring-based communication and reducing latency proportional to the number of GPUs.
2Reliability
If synchronization mechanisms are implemented at every ring step, then data validity is ensured, but the latency increases significantly especially for small message sizes
Solution Approach 1:
The patent extracts the synchronization mechanism from every communication step and consolidates it into a single wave completion event. Instead of fencing and flag-writing at each ring step, the system uses credit-based flow control where the endpoint only needs to synchronize when the entire wave of collective communication primitives is complete, dramatically reducing synchronization overhead while maintaining data validity.
Solution Approach 2:
The patent implements preliminary credit allocation before wave injection, allowing the network to pre-allocate resources and buffer space for incoming communication waves. This preliminary action enables the network elements to process multiple waves in parallel without requiring synchronization at each step, as credits act as advance permission for data injection and consumption.
3Productivity
If more GPUs are added to increase compute load, then processing capacity increases, but communication latency quickly becomes a bottleneck
Solution Approach 1:
The patent moves communication operations from the endpoint dimension into the network dimension by enabling in-network computation. Network elements perform reduction operations directly in the fabric, transforming the communication pattern from endpoint-centric sequential operations to network-centric parallel operations. This dimensional shift allows compute load to scale with GPU count while communication latency remains bounded by network traversal time rather than synchronization overhead.
Data Source
AI summary
A network device configured to perform scalable, in-network computations is described. The network device is configured to process pull requests and/or push requests from a plurality of endpoints connected to the network. A collective communication primitive from a particular endpoint can be received at a network device. The collective communication primitive is associated with a multicast region of a shared global address space and is mapped to a plurality of participating endpoints. The network device is configured to perform an in-network computation based on information received from the participating endpoints before forwarding a response to the collective communication primitive back to one or more of the participating endpoints. An injection policy comprising the issuing of credits enables each endpoint to limit the amount of collective communication primitives injected into the network simultaneously to reduce network congestion caused by increased network traffic due to the multicast capability of the network devices.