Machine-Exception Handling for Distributed Training Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High training costs in machine learning processes due to low computing or communication speeds of some machines in a cluster, leading to slow training processes and increased resource requirements.
Innovation Solution
Implementing machine-exception handling methods to identify abnormal machines by comparing gradient consumption times with average consumption times across the cluster, and adjusting the learning rate to determine when to perform weight updates, thereby optimizing the training strategy and reducing costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If machines with high computing speeds and high communication efficiency are used, then training speed is improved, but machine resource requirements and costs increase
Solution Approach 1:
The patent extracts and removes abnormal machines from the training cluster by detecting machines with gradient consumption times significantly different from the average. This allows the system to continue training with the remaining normal machines, avoiding the need to discard entire machine resources when some machines are abnormal.
Solution Approach 2:
The patent changes the parameter of gradient consumption time threshold to identify abnormal machines. By comparing each machine's gradient consumption time against a calculated threshold (average consumption time multiplied by a coefficient), the system dynamically determines which machines to exclude from training, optimizing the balance between training speed and machine resource utilization.
2Power
If machines with very high loads are used, then computing capacity is improved, but computing speed decreases
Solution Approach 1:
The patent implements feedback by continuously monitoring gradient consumption time for each machine and comparing it against the average. This feedback mechanism allows the system to identify machines that are overloaded or malfunctioning and exclude them from training, ensuring that only machines operating within normal performance ranges participate in the training process.
3Quantity of substance
If machines with low communication speeds are used, then hardware cost is reduced, but training process becomes very slow
Solution Approach 1:
The patent extracts machines with abnormally low communication speeds from the training cluster by detecting them through gradient consumption time analysis. This allows the system to maintain cost-effectiveness by using lower-cost machines while excluding only the problematic ones that would severely slow down the training process.
Data Source
AI summary
The present disclosure provides machine-exception handling methods and learning rate adjustment methods and apparatuses. One exemplary method comprises: acquiring a gradient consumption time of a target machine, wherein the gradient consumption time is used to indicate a gradient related time consumed by the target machine in a training process; determining whether the gradient consumption time satisfies a predetermined condition compared with a pre-acquired average consumption time, wherein the average consumption time is used to indicate an average value of the gradient related time consumed by all machines other than the target machine in a cluster in the training process; and determining that the target machine is abnormal if the gradient consumption time satisfies the predetermined condition compared with the average consumption time. The present disclosure addresses the technical problem of high training costs caused by low computation or communication speeds of some machines in a cluster.