Distributed Parameter Server Model Training Load Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed machine learning systems, the master node becomes a bottleneck due to its heavy workload in model updating and parameter delivery during training, leading to inefficiencies and potential congestion.
Innovation Solution
The method involves splitting the model into sub-models and distributing them across multiple parameter servers in slave nodes, where parameter clients execute training tasks and update sub-models independently, shifting the model updating work from the master node to the parameter servers, allowing for parallel training and reducing the master node's load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the master node performs model updating and parameter delivery for multiple rounds, then the model training can be completed, but the workload of the master node becomes heavy and it becomes a bottleneck
Solution Approach 1:
The patent divides the model into multiple sub-models and distributes them across different parameter servers. Each parameter server independently handles model updates for its assigned sub-model, eliminating the need for the master node to perform all model updating operations. This segmentation of the model and its update responsibilities directly reduces the master node's workload while maintaining complete model training functionality.
Solution Approach 2:
The patent extracts the model updating and parameter delivery functions from the master node and relocates them to distributed parameter servers. Each parameter server is responsible for updating its own sub-model based on training results from parameter clients, thereby removing the bottleneck of centralized model management while preserving the training process integrity.
2Manufacturing precision
If multiple rounds of iterative computation are performed, then the model training accuracy is improved, but the training time increases
Solution Approach 1:
The patent segments the model training process into parallel operations across multiple parameter servers. Each parameter server independently performs iterative computation and model updates for its sub-model simultaneously with other parameter servers. This parallelization maintains the required number of iterative rounds for accuracy while reducing the overall wall-clock training time through concurrent execution.
Solution Approach 2:
The patent enables continuous model training by allowing parameter servers to perform iterative computations and updates without waiting for centralized coordination after each round. Parameter clients continuously send training results to parameter servers, which continuously update sub-models, maintaining an ongoing training process that improves efficiency while preserving accuracy through sufficient iteration rounds.
Data Source
Figure 1A
Figure 1B
Figure 2~3
AI summary
A method and a device for training a model in a distributed system are disclosed, so as to reduce load of a master node (101) during model training. The method includes: receiving, by a parameter server (1022) in a first slave node (102), a training result sent by a parameter client (1021) in at least one slave node (102) in the distributed system, where the first slave node (102) is any slave node (102) in the distributed system, and a parameter client (1021) in each slave node (102) obtains a training result by executing a training task corresponding to a sub-model stored on a parameter server (1022) in the slave node (102); and updating, by the parameter server (1022) in the first slave node (102) based on the received training result, a sub-model stored on the parameter server in the first slave node.