Asynchronous Parameter Aggregation for Distributed ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training machine learning models face challenges due to privacy and connectivity concerns, as well as bandwidth limitations, which prevent the sharing of large datasets across devices, hindering the effectiveness of distributed computing for tasks like navigation and mapping services.
Innovation Solution
A decentralized system where devices train machine learning models locally using asynchronous communication with a parameter server, sharing only model parameters rather than data, allowing for efficient training without centralizing sensitive information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is shared between devices for distributed machine learning, then training effectiveness is improved, but privacy concerns and bandwidth consumption increase
Solution Approach 1:
The patent extracts only the essential model parameters from the complete dataset, transmitting only these compressed representations between devices. This allows training effectiveness to improve through parameter sharing while privacy is protected because the actual sensitive data remains localized and is never transmitted.
Solution Approach 2:
The patent introduces an intermediary mechanism where model parameters act as a mediator between the raw data and the training process. Instead of sharing raw data directly, devices share processed parameter representations that convey training information while obscuring the original sensitive data, thus resolving the privacy-bandwidth contradiction.
2Productivity
If data is shared between devices for distributed machine learning, then training effectiveness is improved, but bandwidth consumption increases
Solution Approach 1:
The patent extracts only the essential model parameters from the complete dataset, transmitting only these compressed representations between devices. This allows training effectiveness to improve through parameter sharing while bandwidth consumption is reduced because only essential parameter information is transmitted rather than complete datasets.
3Stability of the object's composition
If devices are required to be continuously connected for data sharing, then data consistency is improved, but device availability and privacy protection worsen
Solution Approach 1:
The patent implements a dynamic training approach where devices can join and leave the distributed training process at different times. The system adapts to varying device availability by allowing asynchronous participation, eliminating the need for continuous connectivity while maintaining training progress through flexible device engagement.
4Measurement precision
If complete datasets are transmitted between devices, then training accuracy is improved, but transmission time and bandwidth consumption increase
Solution Approach 1:
The patent extracts only the essential model parameters from the complete dataset, transmitting only these compressed representations between devices. This allows training accuracy to improve through parameter sharing while transmission time is reduced because only essential parameter information is transmitted rather than complete datasets.
Data Source
AI summary
Systems and methods are provided for training a machine learned model on a large number of devices, each device acquiring a local set of training data without sharing data sets across devices. The devices train the model on the respective device's set of training data. The devices communicate a parameter vector from the trained model asynchronously with a parameter server. The parameter server updates a master parameter vector and transmits the master parameter vector to the respective device.