Federated Learning Model Aggregation Using Uncertainty Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning approaches, such as Federated Averaging, underestimate the heterogeneity of data with diverse features, leading to misleading interpretations and reduced statistical power when combining data from different sources without accounting for variation across sources, particularly in complex domains with non-identical and non-independent data distributions.
Innovation Solution
The method involves using the uncentered variance of the stochastic gradient as a training parameter to estimate individual intra-variability during training and compute a weighted average of models, allowing for the determination of relative contributions of local datasets to the global model, thereby penalizing uncertainty and improving the robustness of the aggregated model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If simple averaging of local models is used in federated learning, then communication efficiency is improved, but the heterogeneity of diverse data sources is underestimated leading to reduced statistical power
Solution Approach 1:
The patent transforms the aggregation process by changing the parameters used for combining local models. Instead of simple averaging, it uses uncertainty estimates (variance of stochastic gradients) as weighting parameters to determine the contribution of each local model. This parameter change allows the system to account for data heterogeneity while maintaining communication efficiency, as the uncertainty parameters are computed locally and only the aggregated model and uncertainty estimates need to be exchanged.
2Productivity
If local datasets are combined without accounting for variation across sources, then data aggregation is achieved, but misleading interpretations result due to unaccounted heterogeneity
Solution Approach 1:
The patent implements a feedback mechanism where uncertainty estimates from local training processes are collected and used to adjust the aggregation process. The variance of stochastic gradients computed at each client provides feedback about the local data characteristics, which is then used by the server to weight the contribution of each local model appropriately. This feedback loop ensures that data heterogeneity information is preserved and utilized in the global model training.
3Reliability
If weighted average of models is computed using uncertainty estimates, then the robustness of aggregated model is improved, but the complexity of the aggregation process increases
Solution Approach 1:
The patent introduces uncertainty estimates (variance of stochastic gradients) as intermediary variables that mediate between local model training and global aggregation. These intermediaries carry information about local data heterogeneity without requiring direct comparison or sharing of raw data. The intermediary parameters simplify the aggregation process by providing a clear weighting mechanism based on local uncertainty, making the process more manageable despite the increased robustness requirements.
Data Source
AI summary
A method and system are disclosed for providing an aggregated trained machine learning model for performing a prediction task. A main processing device obtains from a first processing device at least a portion of a first trained model having been generated by training an initial model on a first training dataset, and a first training parameter indicative of a level of predictive uncertainty thereof. The main processing device obtains from a second processing device at least a portion of a second trained model having been generated by training the initial model on a second training dataset, and a second training parameter indicative of a level of predictive uncertainty thereof. The main processing device combines, using the first and second training parameters, at least the portion of the first and second trained models to thereby obtain the aggregated trained model. The main processing device provides the aggregated trained model.