Parameter Server RDMA Shared Memory for Distributed Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed deep-learning platforms face inefficiencies in parameter sharing due to high communication overhead and resource idle time caused by message-based parameter exchange between parameter servers and distributed computers, leading to decreased resource utilization.
Innovation Solution
Implementing shared memory across a high-speed network that supports Remote Direct Memory Access (RDMA) for parameter servers and distributed deep-learning processes, allowing direct reading and writing of parameters, thereby reducing memory copies and communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If message-based parameter exchange is used between parameter server and distributed computers, then parameter sharing can be implemented, but communication overhead increases and resource utilization decreases
Solution Approach 1:
The patent extracts the parameter data from message-based communication protocols and places it directly into shared memory regions. Instead of wrapping parameters in communication messages with protocol overhead, the parameters are directly accessible in memory spaces that distributed processes can read and write without protocol processing, thereby removing the harmful communication overhead while preserving parameter sharing capability
Solution Approach 2:
The patent replaces the mechanical message-passing system with a memory-based direct access system. Instead of using software message queues and protocol processing layers, the system uses shared memory regions that allow direct reading and writing of parameters, substituting the message-based mechanical system with a more efficient memory-based access mechanism
2Reliability
If message-based parameter exchange is used between parameter server and distributed computers, then parameter sharing can be implemented, but resource utilization decreases due to CPU/GPU idle time
Solution Approach 1:
The patent enables continuous parameter access by maintaining parameters in shared memory regions that distributed processes can read and write at any time without waiting for message transmission cycles. This eliminates the idle periods where CPUs and GPUs would wait for parameter updates, allowing computational work to proceed continuously while parameters are updated in the background in shared memory
Solution Approach 2:
The patent pre-allocates shared memory regions for parameter storage before distributed training begins. By having parameter memory spaces ready in advance and accessible to all processes, the system eliminates the need for on-the-fly message creation and transmission during training iterations, allowing computations to proceed without waiting for parameter exchange operations
3Stability of the object's composition
If centralized parameter storage is used with synchronous updates, then parameter consistency is maintained, but training speed decreases due to synchronization requirements
Solution Approach 1:
The patent introduces shared memory regions as intermediaries between the parameter server and distributed worker processes. These shared memory regions act as buffer zones where parameters can be updated by the parameter server and subsequently read by worker processes at their own pace, mediating between the need for parameter consistency and the need for asynchronous training speed
4Ease of operation
If parameters are transmitted using communication protocols, then structured data exchange is achieved, but additional memory copies are required increasing overhead
Solution Approach 1:
The patent merges the parameter storage location with the computation location by using shared memory regions that are directly accessible from both the parameter server and worker process memory spaces. This eliminates the need for separate message buffers and intermediate memory copies, as parameters reside in memory regions that can be directly accessed by computational processes, combining storage and computation memory spaces
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly reduces communication overhead and improves resource utilization by enabling direct, high-speed parameter sharing between parameter servers and distributed processes, enhancing the efficiency of distributed deep-learning training.
Implementation Method 1
shared memory across a high-speed network that supports Remote Direct Memory Access (RDMA) for parameter servers and distributed deep-learning processes
Data Source
AI summary
Disclosed herein are a parameter server and a method for sharing distributed deep-learning parameters using the parameter server. The method for sharing distributed deep-learning parameters using the parameter server includes initializing a global weight parameter in response to an initialization request by a master process; performing an update by receiving a learned local gradient parameter from the worker process, which performs deep-learning training after updating a local weight parameter using the global weight parameter; accumulating the gradient parameters in response to a request by the master process; and performing an update by receiving the global weight parameter from the master process that calculates the global weight parameter using the accumulated gradient parameters of the one or more worker processes.