Distributed Training Fault Recovery Using Rescheduled Chips
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current fault processing solutions in AI model training systems require a long recovery time and are inefficient, especially when training nodes or chips fail, leading to interruptions in distributed training tasks.
Innovation Solution
A method where a first chip on the host side continues executing a first subtask while a second chip on the device side saves a fault file, which is then synchronized to a rescheduled chip to continue the second subtask, allowing the execution result of the first subtask to be reused for recovery, thus shortening recovery time and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a fault occurs in distributed training, the entire training task is interrupted to ensure data consistency, then reliability is improved, but productivity deteriorates due to complete task interruption and long recovery time
Solution Approach 1:
The training task is divided into multiple subtasks (first subtask on host side, second subtask on device side). When a fault occurs, only the affected second subtask is interrupted, while the first subtask can continue or be quickly recovered. This segmentation allows partial task continuation, improving recovery efficiency while maintaining data consistency through selective interruption.
Solution Approach 2:
The host side prepares and stores execution results of the first subtask in advance before the fault occurs. When a fault is detected, these pre-prepared results can be quickly transferred to the device side for resuming the second subtask, eliminating the need to re-execute the first subtask and significantly reducing recovery time.
2Reliability
If the host side stops executing the first subtask when a fault occurs, then reliability is improved by ensuring consistent state, but productivity deteriorates due to loss of computation resources and longer recovery time
Solution Approach 1:
The host side continues executing or has already completed the first subtask and stored its results before the fault occurs. This preliminary completion and storage of results allows the device side to quickly resume the second subtask after fault recovery without waiting for the host to re-execute, significantly reducing recovery time while maintaining state consistency through result transfer.
Solution Approach 2:
The host side maintains continuous execution of the first subtask or keeps computation resources active rather than stopping completely. This continuity preserves computation progress and allows rapid resumption or result transfer, reducing the loss of time while ensuring that the system can recover to a consistent state through coordinated resumption of subtasks.
3Reliability
If the device side waits for the host side to complete the first subtask before resuming, then reliability is improved by ensuring data availability, but productivity deteriorates due to idle time on the device side
Solution Approach 1:
The host side pre-computes and stores the results of the first subtask before the fault occurs or during the fault handling process. When the device side needs to resume the second subtask, these results are already available for immediate transfer and use, eliminating the need for the device to wait idle and significantly improving task execution efficiency while ensuring data availability.
Solution Approach 2:
The system implements a feedback mechanism where the host side monitors the completion status of the first subtask and proactively transfers results to the device side when ready. This feedback loop allows the device side to resume execution as soon as data is available, minimizing idle time and improving overall productivity while maintaining reliability through coordinated data transfer.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
This application provides a fault processing method, applied to a training system. The training system includes a first chip on a host side and a plurality of second chips on a device side, and the first chip and the plurality of second chips are configured to collaboratively execute a training task. The training task includes a first subtask and a plurality of second subtasks, and execution of the second subtask depends on an execution result of the first subtask. The method includes: The first chip executes the first subtask. When a fault occurs, the first chip saves a fault file before a second chip in a normal state on the device side stops executing the second subtask. The first chip synchronizes the fault file to a rescheduled chip on the device side, so that the rescheduled chip continues to execute the second subtask. In the method, the first chip on the host side may not stop executing the second subtask. In this way, the execution result of the first subtask on the host side can be reused to recover the training task, to shorten recovery time of the training task, and improve recovery efficiency of the training task.