In-Network Aggregation Switches for ML Training Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning (ML) training systems using parameter server architectures face challenges in efficiently managing network bandwidth and reducing latency, particularly in collective operations like Allgather, where network bottlenecks can occur.
Innovation Solution
The proposed solution involves utilizing network interface devices, such as switches, to perform in-network aggregation using a combination of reliable and non-reliable transport protocols. These devices aggregate data from worker nodes using operations like SUM, SUBTRACT, MIN, MAX, MULTIPLY, and perform floating-point operations, while reducing memory and processor usage by not storing connection state for upstream switches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If parameter servers perform aggregation operations, then computation capability is improved, but network bandwidth utilization deteriorates due to bottlenecks
Solution Approach 1:
The patent segments the aggregation function across multiple network switches rather than concentrating it in a single parameter server. Each switch performs partial aggregation operations on data received from worker nodes, dividing the computational load and distributing it across the network infrastructure. This segmentation prevents any single node from becoming a bottleneck while maintaining overall computation capability.
Solution Approach 2:
The patent moves aggregation operations from the application layer (parameter servers) to the network layer (switches). By implementing in-network aggregation, the system utilizes the network infrastructure itself as a computational resource, adding a new dimension of parallelism across multiple switches simultaneously performing aggregation operations, thereby improving both computation capability and bandwidth utilization.
2Device complexity
If all data is transmitted through parameter servers, then centralized control is improved, but latency increases due to network bottlenecks
Solution Approach 1:
The patent segments the data aggregation process across multiple switches in the network fabric. Instead of all data converging at a single parameter server, intermediate switches perform aggregation operations on subsets of data, reducing the volume of traffic that must traverse the entire network path. This segmentation maintains centralized control through coordinated switch operations while significantly reducing end-to-end latency.
Solution Approach 2:
The patent introduces network switches as intermediary devices that perform aggregation operations between worker nodes and parameter servers. These switches act as mediators that reduce the communication burden on both endpoints by pre-aggregating data locally, thereby reducing latency without eliminating centralized control architecture.
3Device complexity
If worker nodes communicate directly with parameter servers, then system simplicity is improved, but network efficiency deteriorates
Solution Approach 1:
The patent enables network switches to perform multiple functions: traditional packet forwarding and new in-network aggregation operations. By making switches multi-functional, the system improves network efficiency without adding separate dedicated aggregation hardware, maintaining system simplicity while enabling sophisticated data processing capabilities within the network fabric.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Examples described herein relate to a switch comprising circuitry configured to for packet communications associated with a collective operation to train machine learning (ML) models: utilize a reliable transport protocol for communications from at least one worker node of the collective operation to a switch, wherein the utilize a reliable transport protocol for communications from at least one worker node of the collective operation to the switch comprises store packet receipt state for per-packet communications from the at least one worker node of the collective operation to the switch and utilize a non-reliable transport protocol by the switch to a device that is to perform aggregation of results, wherein the reliable transport protocol comprises a different protocol than that of the non-reliable transport protocol.