Distributed Training Node Selection via Waiting and Execution Time Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed training methods for multiple neural networks face challenges in determining the optimal degree of parallelism to minimize communication costs and resource waiting times, especially in multi-tenant environments where resource availability and usage fees are variable.
Innovation Solution
A computer-readable recording medium stores an information processing program that manages distributed training across multiple nodes by calculating scores based on waiting and execution times for different execution environments, and determines the optimal number of nodes to use for training based on these scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of nodes is increased to improve parallelism and reduce execution time, then productivity is improved, but resource waiting time increases due to resource availability constraints in multi-tenant environments
Solution Approach 1:
The system performs preliminary evaluation of multiple execution environments by calculating both waiting time and execution time for different node configurations before actual distributed training begins. This advance assessment allows selection of the optimal node number that minimizes total time (waiting + execution), rather than simply maximizing parallelism without considering resource availability constraints
Solution Approach 2:
The system changes the parameter of node number across multiple execution environments (e.g., 1 node, 2 nodes, 4 nodes, 8 nodes) and evaluates the total time for each configuration. By varying this key parameter and comparing results, the system identifies the optimal point where increased parallelism no longer provides net benefit due to waiting time penalties
2Productivity
If the number of nodes is increased to improve parallelism, then productivity is improved, but device complexity increases due to coordination and communication overhead
Solution Approach 1:
The system applies partial action by selecting an optimal subset of nodes rather than using all available nodes. The evaluation process determines the point of diminishing returns where adding more nodes creates excessive coordination overhead that outweighs the parallelism benefits, thus using only the necessary number of nodes for the specific training task
3Productivity
If the number of nodes is increased to improve parallelism, then productivity is improved, but cost increases due to resource usage fees in multi-tenant environments
Solution Approach 1:
The system performs preliminary cost-benefit analysis by evaluating execution time and associated costs for different node configurations before committing resources. This advance evaluation prevents wasteful allocation of excessive nodes that would incur unnecessary usage fees while still achieving acceptable training performance
Solution Approach 2:
The system varies the node number parameter across multiple execution environments and evaluates the cost-effectiveness of each configuration. By comparing total time and cost across different node counts, the system identifies the optimal point that achieves sufficient parallelism without incurring excessive resource usage fees
Data Source
AI summary
A non-transitory computer-readable recording medium stores an information processing program for causing a processor of an information processing apparatus that manages distributed training that uses a plurality of nodes to execute a process. The process includes: obtaining a waiting time until a resource to be used for the distributed training is secured and an execution time taken for the distributed training, for each of execution environments of different numbers of nodes; obtaining a score for each of the execution environments based on the waiting time and the execution time acquired for each of the execution environments; and determining the number of nodes to be used for the distributed training based on a plurality of the scores.