Federated Deep Learning Weight Aggregation for Non-IID Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing federated learning methods, such as FedAvg, FedProx, and SCAFFOLD, struggle with non-IID data, leading to suboptimal performance and convergence issues in training deep learning models across decentralized entities.
Innovation Solution
A method involving partitioning local datasets into sub-datasets, training deep learning models on these sub-datasets, aggregating weights at a central server using a regression model, and iteratively refining the global model weights to improve convergence and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If FedAvg algorithm is used to aggregate client weights, then communication efficiency is maintained, but model performance deteriorates due to non-IID data
Solution Approach 1:
The patent segments the aggregation process into multiple components: local model training at each client, weight compression to reduce communication overhead, and server-side aggregation that combines weights from multiple clients. This segmentation allows efficient communication while maintaining model performance through distributed training.
Solution Approach 2:
The patent applies local quality by allowing each client to train models with their own local data distributions (non-IID conditions) while maintaining a global model structure. Each client's local model adapts to its specific data characteristics, and the server aggregates these localized improvements to enhance overall model performance.
2Reliability
If FedProx algorithm is used to limit distance between local and global models, then convergence is improved, but communication size doubles due to control variables
Solution Approach 1:
The patent extracts only the essential weight parameters needed for model training and transmits them to the server. By removing unnecessary control variables and focusing on core model weights, the communication overhead is reduced while maintaining convergence properties through efficient weight aggregation.
Solution Approach 2:
Instead of having the server send control variables to clients (as in FedProx), the patent inverts the approach by having clients send their trained weights to the server for aggregation. This reversal eliminates the need for downward control variable transmission while achieving convergence through upward weight feedback.
3Measurement precision
If SCAFFOLD algorithm is used with control variables, then update direction estimation is improved, but communication size per iteration doubles
Solution Approach 1:
The patent uses weight compression techniques to create compact representations of model weights that can be transmitted efficiently. By compressing weight matrices into smaller formats (e.g., using low-rank decomposition or quantization), the system maintains precision in weight transmission while significantly reducing communication size.
Solution Approach 2:
The patent changes the parameter representation format to reduce communication overhead. By transforming weights into compressed forms or alternative parameterizations that convey the same information with fewer bits, the system achieves precise update direction estimation with reduced communication requirements.
4Adaptability or versatility
If FedNova algorithm is used to handle different local steps, then adaptability is improved, but aggregation accuracy deteriorates
Solution Approach 1:
The patent implements dynamic adaptation by allowing each client to perform a different number of local training steps based on their data characteristics and computational resources. The server aggregates weights from clients with varying local steps, adapting the aggregation process to accommodate heterogeneous training dynamics while maintaining accuracy through normalized weight combination.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The disclosure notably relates to a computer-implemented method for federated learning a deep learning model. The method is implemented on a system comprising a central server and n entities. The method comprises, iteratively repeating, until convergence, the following steps: by each entity i, training (S22) the initial deep learning model (M) sent by the central server with the respective dataset (Di) of the entity and with each sub-dataset j, building (S30), by the server, a training dataset of training samples from all the obtained sets of weights, training (S40) a regression model based on the built training dataset, predicting (S50), using the trained regression model, a set of weights (W) and sending (S60) the predicted set of weights (W) to each entity for the next iteration. Such a method forms an improved solution for federated learning a deep learning model.