Shared Memory Gradient Aggregation in Distributed ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Bulk Synchronous Parallel (BSP) systems in machine learning are inefficient due to synchronization barriers, leading to underutilization of resources and slow convergence rates, while asynchronous parallel (AP) and stale synchronous parallel (SSP) systems offer performance advantages but are harder to implement and require specialized systems, limiting scalability and adoption.
Innovation Solution
A computer system that uses a shared memory module to decouple asynchronous gradient computation from synchronous periodic model updates, allowing machine learning modules to operate independently and asynchronously, reducing wait times and enabling efficient data flow between components, while reusing mature BSP systems for rapid implementation and improved scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If BSP systems use synchronization barriers to ensure all workers complete gradient computation before model updates, then model training accuracy is maintained, but system resources are underutilized and convergence rate is slow
Solution Approach 1:
The patent introduces dynamic gradient aggregation where the parameter server accepts and processes gradients asynchronously without waiting for all workers to complete. The system dynamically adjusts to varying worker speeds, allowing faster workers to contribute gradients earlier while slower workers continue computing, thus improving convergence rate without sacrificing training accuracy through the use of gradient accumulation and periodic synchronization.
Solution Approach 2:
The patent eliminates idle waiting time at synchronization barriers by implementing continuous gradient computation and submission. Workers continuously compute and submit gradients without pausing for synchronization barriers, and the parameter server continuously aggregates gradients as they arrive. This continuous operation maximizes resource utilization and accelerates convergence while maintaining accuracy through gradient accumulation mechanisms.
2Manufacturing precision
If BSP systems implement strict synchronization barriers, then gradient merging accuracy is ensured, but wait times increase and resource utilization decreases
Solution Approach 1:
The patent implements gradient accumulation where workers pre-compute multiple gradients locally before submitting to the parameter server. This preliminary action allows gradients to be ready for immediate aggregation without waiting for synchronization barriers, reducing wait time while maintaining merging accuracy through accumulated gradient updates that are periodically synchronized.
Solution Approach 2:
The patent introduces gradient buffers and accumulation mechanisms as intermediaries between workers and the parameter server. These intermediaries temporarily store gradients from multiple workers and perform accurate merging without requiring simultaneous arrival of all gradients, thus eliminating wait times while preserving gradient merging accuracy through buffered aggregation.
3Productivity
If AP systems eliminate synchronization barriers to improve resource utilization, then convergence rate increases, but workers may merge delayed gradients on different models reducing accuracy
Solution Approach 1:
The patent implements version tagging and gradient metadata that provide feedback to the parameter server about which model version each gradient corresponds to. This feedback mechanism ensures that gradients are accurately matched with their corresponding model versions during aggregation, preventing the merging of delayed gradients on different models and maintaining gradient merging accuracy while allowing asynchronous operation.
Solution Approach 2:
The patent creates and maintains copies of the model state with version identifiers at both workers and the parameter server. Each gradient is tagged with the model version it was computed on, and the parameter server uses these version copies to ensure accurate matching and aggregation of gradients with their corresponding model states, preventing accuracy loss from mismatched gradient merging.
4Productivity
If SSP systems bound model staleness with a staleness factor, then convergence rate is improved, but system complexity increases and implementation becomes harder
Solution Approach 1:
The patent uses model version numbers and gradient metadata parameters to track and manage staleness implicitly. Instead of implementing explicit staleness factor control mechanisms, the system changes the parameter representation to include version tags, allowing automatic staleness management through version matching. This approach improves convergence rate by enabling asynchronous operation while avoiding the complexity of explicit staleness factor enforcement.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention provides a computer system (100) for distributed training of a machine learning model. The computer system (100) comprises a BSP system (101) including a central BSP control module (102) and at least one local BSP module (103). It further comprises at least one machine learning module (105) associated with exactly one local BSP module (103), and a shared memory module (104) associated with exactly one pair of a local BSP modul (103) and a machine learning module (105). The central BSP control module (102) is configured to instruct the at least one local BSP module (103) to store, in its associated shared memory module (104), a local model. The at least one machine learning module (105) is configured to read, from its associated shared memory module (104), the local model, compute a gradient based on the local model, and aggregate the gradient immediately after its computation into an aggregated gradient in its associated shared memory module (104). The central BSP control module (102) is further configured to instruct the at least one local BSP module (103) to periodically read out its associated shared memory module (104).