Parallel Stochastic Gradient Descent Model Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for collecting and utilizing data from diverse sources to make predictions are ineffective, leading to inefficiencies in determining the success of courses of action in business and health fields.
Innovation Solution
A system employing parallel implementation of stochastic gradient descent (SGD) to process training datasets across multiple geographic locations, computing local models in parallel and combining them into a global model using symbolic representations, reduces computation time and resource usage while maintaining the sequential dependencies of SGD.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sequential SGD is used to process training datasets, then model parameter computation accuracy is maintained, but computation time increases significantly
Solution Approach 1:
The patent divides the training dataset into multiple partitions distributed across different geographic locations, with each location processing its local partition in parallel to compute local model parameters. This segmentation enables concurrent processing while maintaining the sequential SGD computation logic within each partition, thereby reducing overall computation time without sacrificing model accuracy.
2Productivity
If parallel SGD is implemented across multiple geographic locations, then computation time decreases, but maintaining sequential dependencies becomes complex
Solution Approach 1:
The patent introduces a central server as an intermediary that coordinates parallel SGD computations across multiple geographic locations. The central server manages the distribution of training data partitions, collects local model parameters from various locations, and orchestrates the aggregation process. This intermediary structure simplifies the complexity of direct peer-to-peer coordination while maintaining sequential dependencies through controlled parameter updates.
Solution Approach 2:
The patent combines local model parameters computed in parallel at different geographic locations into a global model parameter set. By merging the results from multiple parallel computations while preserving sequential SGD dependencies through proper aggregation techniques, the system achieves both computational efficiency and model accuracy.
3Measurement precision
If large training datasets are transmitted between locations, then model accuracy improves, but network bandwidth consumption and latency increase
Solution Approach 1:
The patent extracts and processes only the essential model parameters locally at each geographic location rather than transmitting entire training datasets. Each location computes local model parameters from its distributed training data partition and transmits only these compact parameter representations to the central server for aggregation. This extraction approach maintains prediction accuracy while dramatically reducing network bandwidth consumption and latency.
Data Source
AI summary
Described herein is a system that transmits and combines local models, that individually comprise a set of local parameters computed via stochastic gradient descent (SGD), into a global model that comprises a set of global model parameters. The local models are computed in parallel at different geographic locations along with symbolic representations. Network transmission of the local models and the symbolic representations, rather than transmission of the large training data subsets processed to compute the local models and symbolic representations, conserves resources and decreases latency. The global model can then be used as a model to determine a likelihood of a course of action being successful for an organization. For example, the course of action can be a purchase of a security or a business operation strategy. In another example, the course of action can be a type of medical treatment for a patient.