Distributed ML Training Checkpoint Transfer Across Heterogeneous Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, such as neural networks, face challenges in training due to architectural incompatibilities between different computing systems, leading to disruptions in training progress and results when transferred between systems with varying architectures.
Innovation Solution
A method and system for transferring the training of a machine learning model at a check-point from one machine with a specific architecture to another with different resources, using a scheduler machine to manage the process and adjust hyper-parameters for consistency, allowing training on heterogeneous computing systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If training is transferred between computing systems with different architectures, then resource utilization is improved, but training consistency and results deteriorate
Solution Approach 1:
The patent introduces a scheduler machine as an intermediary between the training job and the computing systems. The scheduler machine receives the training job, determines appropriate computing systems, manages checkpoint transfers, and coordinates hyperparameter adjustments. This intermediary layer enables flexible resource utilization across heterogeneous systems while maintaining training consistency through centralized control and coordination.
2Productivity
If training is halted and resumed on different systems, then system availability is improved, but training progress and results deteriorate
Solution Approach 1:
The patent implements preliminary checkpointing where the training system saves intermediate training states (checkpoints) at predetermined intervals or trigger points before halting. These checkpoints include model parameters, optimization state, and hyperparameter configurations. When training resumes on a different system, the scheduler retrieves the appropriate checkpoint and restores the training state, eliminating the need to restart from scratch and preserving training progress despite system transitions.
3Adaptability or versatility
If hyperparameters are adjusted for different systems, then training adaptability is improved, but training precision deteriorates
Solution Approach 1:
The patent systematically adjusts hyperparameters based on the characteristics of the target computing system. The scheduler machine modifies hyperparameters such as batch size, learning rate, and optimization settings according to the computational capabilities, memory constraints, and processing architecture of the available systems. This controlled parameter adaptation enables training to proceed effectively on heterogeneous systems while the scheduler monitors and manages these changes to maintain acceptable training precision.
Data Source
AI summary
In one embodiment, a method includes establishing access to first and second different computing systems. A machine learning model is assigned for training to the first computing system, and the first computing system creates a check-point during training in response to a first predefined triggering event. The check-point may be a record of an execution state in the training of the machine learning model by the first computing system. In response to a second predefined triggering event, the training of the machine learning model on the first computing system is halted, and in response to a third predefined triggering event, the training of the machine learning model is transferred to the second computing system, which continues training the machine learning model starting from the execution state recorded by the check-point.