Distributed Neural Network Training With Worker Failure Redistribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed training of multiple neural networks, failures in one worker lead to training accuracy deterioration and increased training time due to the need for restarting with a proxy worker, especially in congested environments where securing a proxy worker is difficult.
Innovation Solution
Implement a management worker that detects failures and determines whether to continue training with a remaining worker group or secure a proxy worker, distributing the left worker's tasks among the remaining workers to minimize training time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a proxy worker is secured to replace the failed worker, then training reliability is improved, but training time increases and productivity decreases
Solution Approach 1:
The patent applies preliminary action by having the management worker detect worker failures and redistribute tasks before the training process is significantly disrupted. When a worker failure is detected, the system proactively reassigns the failed worker's tasks to remaining workers, allowing training to continue without waiting for a proxy worker to be secured, thus maintaining productivity while ensuring reliability
Solution Approach 2:
The management worker serves as an intermediary that coordinates between failed workers and remaining workers. It detects failures, determines whether to continue training with remaining workers or secure a proxy worker, and distributes tasks accordingly, thereby resolving the contradiction between reliability and productivity
2Measurement precision
If distributed training is restarted with a proxy worker, then training accuracy is improved, but training time increases
Solution Approach 1:
The patent applies dynamics by making the training continuation decision adaptive rather than static. The management worker dynamically determines whether to continue training with remaining workers or secure a proxy worker based on the number of failed workers and system state, allowing the system to optimize between accuracy and time loss in different scenarios
Solution Approach 2:
The system changes the parameter of worker configuration dynamically. When failures occur, the management worker can continue training with a reduced set of remaining workers or introduce a proxy worker, changing the operational parameters to balance training accuracy requirements with time efficiency
3Reliability
If the system waits to secure a proxy worker before continuing training, then training reliability is improved, but training efficiency decreases
Solution Approach 1:
The management worker performs preliminary failure detection and task redistribution before the training process is significantly disrupted. By proactively reassigning tasks from failed workers to remaining workers, the system maintains training efficiency while ensuring reliability through controlled continuation decisions
Data Source
AI summary
A distributed training device includes a processor that executes a procedure. The procedure includes: in distributed training in which a plurality of workers is in charge of training processing of each of a plurality of neural networks of multiple neural networks that integrate inference results of the plurality of neural networks and output a final inference result, detecting whether or not a failure has occurred in each of the plurality of workers, determining, when occurrence of a failure is detected in one or more first workers among the plurality of workers, whether or not to continue the distributed training using a second worker other than first workers among the plurality of workers, and in a case of continuing the distributed training, distributing training processing that the first worker is in charge of to the second worker, and continuing the distributed training.