Distributed Training Fault Recovery with Heartbeat Node Diagnostics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed training of large language models (LLMs) faces challenges in achieving high training efficiency and stability at scale due to failures and stragglers, particularly when using tens of thousands of GPUs, which are computationally intensive and require efficient fault identification and recovery mechanisms.
Innovation Solution
A robust training framework that includes heartbeat monitoring for anomaly detection, self-check diagnostics, and automatic fault recovery by replacing faulty nodes with healthy ones, along with optimized parallelism strategies like data, pipeline, and tensor parallelism to enhance training efficiency and stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed training is scaled to tens of thousands of GPUs, then training capability and model size are improved, but system reliability and fault management become significantly worsened
Solution Approach 1:
The system performs preliminary health checks and diagnostics on all computing nodes before they are assigned to training tasks. Nodes must pass comprehensive self-check diagnostics including hardware status verification, software component validation, and communication tests. This preliminary screening and continuous monitoring enables early detection of potential failures before they impact training stability.
Solution Approach 2:
Each computing node performs self-diagnosis and self-reporting through heartbeat messages and diagnostic tests. The system automatically identifies faulty nodes, determines appropriate replacements, and coordinates recovery without requiring manual intervention. This self-service approach enables the large-scale distributed system to maintain reliability autonomously despite the increased complexity of scaling to tens of thousands of GPUs.
2Reliability
If automatic fault recovery mechanisms are implemented, then training stability is improved, but system complexity and management overhead are worsened
Solution Approach 1:
The system implements continuous feedback loops where computing nodes send heartbeat messages reporting their operational status. The training system monitors these signals and automatically detects abnormalities. When faults are detected, the system receives diagnostic information from nodes, identifies the root cause, and executes recovery procedures. This feedback mechanism simplifies fault management by providing structured, real-time information that enables automated responses without increasing operational complexity.
Solution Approach 2:
The system introduces an intermediary fault detection and management layer that sits between the training tasks and the physical hardware. This intermediate system handles all fault detection, diagnosis, and coordination activities, shielding the training processes from the complexity of underlying hardware management. The intermediary absorbs the complexity of managing tens of thousands of nodes while presenting simple, manageable interfaces to both the training system and administrators.
3Measurement precision
If comprehensive self-check diagnostics are performed on all nodes, then fault detection accuracy is improved, but training time and computational resources are worsened
Solution Approach 1:
The comprehensive diagnostic system is segmented into multiple independent check modules that can be executed selectively. The self-check diagnostics are divided into hardware status verification, software component validation, communication tests, and performance benchmarks. The system performs only the necessary segments based on detected anomalies and training phase requirements, rather than executing all diagnostics continuously, thereby maintaining accuracy while reducing time overhead.
Solution Approach 2:
Instead of performing comprehensive diagnostics continuously, the system implements periodic health checks at strategic training milestones and after node failures. Self-check diagnostics are executed periodically during training operations rather than continuously, with the frequency adjusted based on training phase and detected anomalies. This periodic approach maintains fault detection accuracy while significantly reducing the cumulative time and computational resources required compared to continuous comprehensive checking.
Data Source
AI summary
Example embodiments of the present disclosure relate to a method, a device and a non-transitory computer-readable medium for distributed training. The method comprises obtaining, during a distributed training task performed across a plurality of computing nodes, at least one heartbeat message from the plurality of computing nodes, each computing node including multiple GPU workers; detecting, based on the at least one heartbeat message, an abnormal status of the distributed training task; commanding the plurality of computing nodes to run at least one self-check diagnostics test; identifying, based on results of the at least one self-check diagnostics test, at least one faulty node from the plurality of computing nodes; and replacing the at least one faulty node with an equivalent number of heathy computing nodes that have passed the at least one self-check diagnostics test.