Elastic AI Training on Preemptive Instances Without Task Termination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The instability of preemptive instances in cloud computing platforms used for AI model training leads to increased training costs and compromised training quality due to potential termination of training tasks.
Innovation Solution
Implementing an elastic training framework that dynamically adjusts the number of preemptive instance nodes through elastic scale-out and scale-in based on their availability and use state, allowing seamless integration of on-demand instances when preemptive instances are reclaimed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If preemptive instances are used for AI model training, then training costs are reduced, but training stability deteriorates due to potential instance reclamation
Solution Approach 1:
The system dynamically adjusts the number of worker processes based on instance availability. When preemptive instances are reclaimed, the system automatically scales out by spawning new worker processes on available instances. This dynamic adaptation resolves the contradiction by making the training system flexible enough to handle instance reclamation while maintaining cost efficiency through continued use of preemptive instances.
Solution Approach 2:
The system performs preliminary actions by saving training states and checkpoints before instance reclamation occurs. This allows the training task to be restored and continued on new instances, preventing complete training failure and maintaining stability while still using cost-effective preemptive instances.
2Productivity
If the number of worker processes is increased to maintain training quality, then resource utilization improves, but system complexity increases
Solution Approach 1:
The system implements self-service through automatic worker process management. The training system autonomously monitors instance availability, dynamically spawns or terminates worker processes, and manages state restoration without manual intervention. This self-managing approach maintains high training quality through appropriate resource allocation while minimizing system complexity by eliminating the need for manual configuration and management of worker processes.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
Embodiments of this application provide a training method, a device, a system, and a storage medium. In the training method, a distributed training group executes a training task based on an elastic training framework, so that a worker process of the training task may be run on an instance node of the distributed training group. Because the elastic training framework supports a dynamic change of the worker process of the training task, the training process supports a dynamic quantity change of distributed nodes required for the training. Based on this, when the distributed training group includes a preemptive instance node, elastic scale-out or elastic scale-in may be performed on the training task with reference to states of the preemptive instance node. Further, training may be performed by fully using the preemptive instance, to reduce training costs while enabling the training task to automatically perform scale-out or scale-in instead of terminating, thereby facilitating improvement of training quality and training efficiency.