Parallel SGD Model Combiners for Communication Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current parallel stochastic gradient descent (SGD) algorithms face scalability limitations due to frequent communication, which increases communication cost and reduces accuracy, especially in sparse datasets, as they struggle to maintain inter-step dependence between model updates.
Innovation Solution
The proposed solution involves a parallel SGD algorithm that uses model combiners to periodically update the global model, allowing threads to communicate less frequently while achieving a high-fidelity approximation of sequential processing by accounting for changes in the global model using a symbolic model combiner, enabling efficient combination of local models into a new global model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If frequent communication is used to maintain inter-step dependence between model updates, then model accuracy is preserved, but communication cost increases and scalability decreases
Solution Approach 1:
The patent applies preliminary action by having each thread pre-compute a model combiner before parallel processing begins. This combiner is calculated based on the thread's local data characteristics and the global model structure, allowing the thread to independently adjust its updates without frequent communication. The combiner is prepared in advance to account for inter-step dependence, enabling accurate model aggregation while reducing communication frequency during the actual parallel training process.
Solution Approach 2:
The patent introduces model combiners as intermediaries between local thread models and the global model. Instead of direct frequent communication between threads and the central model, the combiner acts as a mediator that translates local updates into globally consistent adjustments. This intermediary mechanism preserves the inter-step dependence relationship while enabling less frequent communication, as the combiner encapsulates the necessary adjustment information.
2Productivity
If communication frequency is reduced to improve scalability, then communication cost decreases, but model accuracy deteriorates due to increased staleness
Solution Approach 1:
The patent applies local quality by customizing model combiners for each individual thread based on its specific data characteristics and processing patterns. Instead of using a uniform communication strategy for all threads, each thread receives a tailored combiner that accounts for its local data distribution and update characteristics. This localized approach allows each thread to maintain accurate updates with less frequent communication, as its combiner is specifically designed to handle its local staleness issues.
Solution Approach 2:
The patent changes parameters by dynamically adjusting model combiners based on the global model's evolving state. The combiners are not fixed but are recalculated or adjusted as the global model changes, allowing the system to adapt to increasing staleness over time. This parameter change mechanism ensures that even with reduced communication frequency, the combiners remain effective at translating local updates into accurate global model improvements.
3Reliability
If sequential processing is used to maintain inter-step dependence, then model accuracy is maintained, but processing speed decreases
Solution Approach 1:
The patent applies segmentation by dividing the model training process into independent thread segments, each with its own model combiner. This allows multiple threads to process data in parallel while the combiner ensures that inter-step dependence is maintained during aggregation. The segmentation enables concurrent processing of different data portions without sacrificing the sequential accuracy guarantees that would otherwise be required.
Solution Approach 2:
The patent implements feedback by using model combiners that incorporate information about the global model's state and the thread's local processing characteristics. The combiner provides feedback mechanisms that adjust local updates based on global context, ensuring that parallel processing maintains the same accuracy properties as sequential processing. This feedback loop allows the system to achieve sequential-level accuracy with parallel processing speed.
Data Source
AI summary
Systems, methods, and computer-readable media are disclosed for parallel stochastic gradient descent using linear and non-linear activation functions. One method includes: receiving a set of input examples; receiving a global model; and learning a new global model based on the global model and the set of input examples by iteratively performing the following steps: computing a plurality of local models having a plurality of model parameters based on the global model and at least a portion of the set of input examples; computing, for each local model, a corresponding model combiner based on the global model and at least a portion of the set of input examples; and combining the plurality of local models into the new global model based on the current global model and the plurality of corresponding model combiners.