Distributed Deep Learning Model Training via Local Weight Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Centralized deep learning training platforms are computationally expensive, require significant data communication bandwidth, and pose privacy risks due to the need for local devices to share raw input data with a central server.
Innovation Solution
A distributed training platform where local devices receive input data, locally label it, and train a deep learning model without sending raw data to a central server. Instead, they transmit their trained model weights to a host server, which aggregates these weights to approximate a global deep learning model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If centralized deep learning training platform is used, then global deep learning model can be trained, but computational cost increases significantly
Solution Approach 1:
The training process is segmented into distributed tasks where multiple local devices independently train local deep learning models using local data. Each local device performs training operations locally rather than centralizing all computation at a single server, thereby reducing the computational burden on any single system while collectively achieving global model training.
2Productivity
If centralized training platform is used, then model training can be performed, but data communication bandwidth requirement increases
Solution Approach 1:
The raw input data is extracted from the central server and distributed to local devices for local processing. Only the trained model weights (which are much smaller in size compared to raw data) are transmitted back to the central server for aggregation, significantly reducing the quantity of data that needs to be communicated over the network during the training process.
3Productivity
If centralized training platform is used, then deep learning model can be trained, but privacy risk increases due to raw data sharing
Solution Approach 1:
Local devices perform self-service by training their own deep learning models using their local data independently. The raw input data remains on local devices and is not shared with the central server. Only the derived model weights are transmitted for aggregation, allowing each local device to maintain privacy while still contributing to the overall training objective.
4Use of energy by moving object
If distributed training platform is used, then computational cost is reduced, but system complexity increases
Solution Approach 1:
The system employs a universal aggregation server that can receive model weights from multiple local devices with different hardware configurations and data characteristics. The server performs unified weight aggregation to produce a global model that works across all local devices, providing a single point of coordination that simplifies the overall system architecture despite the distributed nature of computation.
Data Source
AI summary
Systems and methods for distributed training of deep learning models are disclosed. An example local device to train deep learning models includes a reference generator to label input data received at the local device to generate training data, a trainer to train a local deep learning model and to transmit the local deep learning model to a server that is to receive a plurality of local deep learning models from a plurality of local devices, the server to determine a set of weights for a global deep learning model, and an updater to update the local deep learning model based on the set of weights received from the server.