Gradient Vector Reduction in Degraded ML Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning model training systems face challenges in efficiently processing large datasets when nodes in the replica network are degraded due to hardware or software issues, leading to impaired performance and potential further degradation from overheating caused by naive data rerouting.
Innovation Solution
A training system that routes and distributes computation to non-degraded nodes efficiently, avoiding further degradation by designating forwarding and critical groups within the replica network, allowing for parallel processing and gradient vector reduction without physical modifications, and supporting both mesh and torus topologies of arbitrary dimension.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data traffic is rerouted around degraded nodes using conventional approaches, then the training can continue despite node failures, but overheating occurs at regions of the replica network causing further degradation
Solution Approach 1:
The patent designates specific nodes as forwarding group members versus critical group members based on their local network conditions and degradation status. This local differentiation allows traffic to be routed through appropriate nodes without concentrating all traffic on a single path, thereby preventing overheating at specific regions while maintaining training continuity.
2Reliability
If computation is concentrated on non-degraded nodes to maintain training, then processing can continue, but resource efficiency decreases due to uneven workload distribution
Solution Approach 1:
The patent segments the replica network into forwarding group members and critical group members. This segmentation allows the system to maintain training continuity by routing through non-degraded nodes while preserving resource efficiency by distributing computation tasks appropriately across available nodes based on their capacity and status.
3Reliability
If naive rerouting is performed around degraded nodes, then training can proceed, but performance degrades due to suboptimal routing paths
Solution Approach 1:
The patent performs preliminary designation of forwarding group members and critical group members before training operations commence. This preliminary action establishes optimal routing paths in advance, preventing performance degradation during actual training by ensuring data traffic flows through the most efficient available paths rather than attempting reroutes during operation.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Methods, systems, and apparatus, including instructions encoded on storage media, for performing reduction of gradient vectors for a network having one or more degraded nodes. A method comprises training a respective replica of a machine learning model on each node of multiple nodes organized in an n-dimensional network topology, combining the respective individual gradient vectors in the nodes to generate a final gradient vector by performing operations comprising: designating each group of nodes along the dimension as either a forwarding group or a critical group, updating, for each receiving node, a respective individual gradient vector with an intermediate gradient vector, performing a reduction on each critical group of nodes along the dimension to generate a respective partial final gradient vector for the critical group, and updating, for each critical group of nodes, an individual gradient vector for a representative node with the respective partial final gradient vector.