Distributed Neural Network Training on Edge Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training neural networks with distributed datasets is challenging due to data privacy constraints, high communication and computational costs, and the need for centralized processing power, which is often prohibited by regulations and expensive to maintain.
Innovation Solution
A distributed neural network architecture that splits the neural network across multiple edge devices, allowing for training on vertically or horizontally distributed data without transferring raw data, using a pipeline to identify common identifiers and perform privacy-preserving operations, thus reducing communication and computational costs while maintaining data privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred to a centralized location for neural network training, then processing power and training efficiency are improved, but data privacy constraints and communication costs worsen
Solution Approach 1:
The neural network is segmented into multiple portions that are distributed across different edge devices. Each device holds and processes data locally without transferring it to a centralized location. The segmented network portions collaborate to achieve joint training objectives while maintaining data privacy through local processing.
Solution Approach 2:
An intermediary coordination mechanism is introduced to manage communication between distributed network portions. This intermediary enables the exchange of necessary information (such as gradient updates or parameter adjustments) without requiring direct data transfer between devices or to a centralized server, thus maintaining privacy while enabling collaboration.
2Power
If a centralized location is used for processing large datasets, then computational power is improved, but cost and infrastructure requirements worsen
Solution Approach 1:
The computational workload is segmented and distributed across multiple edge devices instead of concentrating it in a single centralized system. Each device performs local computations on its own data, reducing the need for expensive centralized hardware infrastructure while maintaining overall computational capability.
Solution Approach 2:
Each edge device performs self-service computation by processing data locally using its own computational resources. This eliminates the need for centralized processing infrastructure, as each device independently contributes its computational power to the distributed training process.
3Object-affected harmful factors
If federated averaging is used for distributed datasets, then data privacy is improved, but communication and computational costs worsen
Solution Approach 1:
The neural network is divided into multiple portions that can be processed independently at different edge devices. This segmentation reduces the amount of data and computation that needs to be exchanged during training, as each device works on its own network portion rather than requiring full model synchronization at every step.
Solution Approach 2:
Instead of performing complete model averaging across all devices at every training iteration (excessive action), the system performs partial updates and optimizations locally at each device. This partial action approach reduces communication frequency and computational overhead while still achieving effective distributed training.
Data Source
AI summary
One example method includes a pipeline for a distributed neural network. The pipeline includes a first phase that identifies intersecting identifiers across datasets of multiple clients in a privacy preserving manner. The second phase includes a distributed neural network that includes a data receiving portion at each of the clients and an orchestrator portion at an orchestrator. The data receiving portions and the orchestrator portions communicate forward and backward passes to perform training without revealing the raw training data.