Client-Side Pre-Training Perturbation to Avoid Federated Learning Local Minima
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning deployments face slower convergence and lower accuracy due to all clients starting training at the same point, leading to susceptibility to local minima.
Innovation Solution
Perform pre-training perturbation by injecting noise into the global model before local training at each trainer client, allowing them to explore different starting points in the learning space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all clients use the same model for each round of training, then model consistency is maintained, but convergence speed decreases and accuracy is reduced due to susceptibility to local minima
Solution Approach 1:
The patent applies parameter changes by introducing perturbation parameters (noise) to the model parameters before local training. Each client receives the global model, adds perturbation to create a perturbed model, trains locally, and returns the updated model. This changes the starting parameters for each client while maintaining overall system consistency through aggregation.
2Stability of the object's composition
If all clients start training at the same point, then initialization consistency is maintained, but model accuracy decreases due to local minima susceptibility
Solution Approach 1:
The patent applies asymmetry by introducing asymmetric perturbation to each client's model parameters. Instead of all clients starting at the same symmetric point, each client receives a uniquely perturbed model that reflects their local data characteristics. This asymmetric initialization allows clients to explore different regions of the parameter space, reducing local minima susceptibility while maintaining overall system coherence through aggregation.
Data Source
AI summary
In one embodiment, a device in a federated learning system receives a global model from an aggregation node. The device applies noise to the global model, to form a noise-augmented model. The device performs local training using the noise-augmented model and a local training dataset, to form a local model. The device provides, via a network, the local model to the aggregation node for aggregation with other local models trained in the federated learning system.