Federated Learning Global Model Weight Extrapolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning systems face challenges in creating a generic global machine learning model when only a small subset of users participate in each iteration, leading to biased models and poor performance, especially in scenarios like the Internet of Senses where data distributions vary significantly.
Innovation Solution
A statistical method is introduced to extrapolate information from non-participating users by constructing probability distributions of model weights from participating users, allowing for representative sampling and updating of the global model to ensure it remains unbiased and generic across all users.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If only a small subset of users participate in each federated learning iteration, then communication overhead and computational cost are reduced, but the global model becomes biased and loses generality across all users
Solution Approach 1:
The patent introduces statistical extrapolation as an intermediary mechanism that bridges the gap between participating and non-participating users. By constructing probability distributions from participating users' model updates and extrapolating to represent all users, the system maintains model generality without requiring full user participation in each iteration
Solution Approach 2:
The patent creates representative copies of non-participating users' model updates by sampling from constructed probability distributions. These synthetic updates mimic what non-participating users would contribute, allowing the global model to remain unbiased and generalizable across the entire user population
2Device complexity
If traditional averaging methods are used to update the global model from subset user updates, then the process is simple and computationally efficient, but the resulting global model is biased towards participating users
Solution Approach 1:
The patent transforms the model update process by changing from direct averaging of participant updates to a statistical extrapolation process. Probability distributions are constructed from participant updates, then sampled to generate representative updates for all users, fundamentally altering how information from non-participating users is represented
Solution Approach 2:
The patent replaces the mechanical averaging operation with a statistical sampling process. Instead of simply averaging available updates, the system uses probability distributions and random sampling to extrapolate and represent the contributions of all users, substituting a more sophisticated statistical mechanism for the simple arithmetic mean
Data Source
AI summary
A method for distributed machine learning (ML) at a central computing device is provided. The method includes: providing a global ML model to a plurality of local computing devices, wherein the global ML model includes a plurality of parameters; receiving, from each local computing device in a subset of the plurality of local computing devices, a local ML model updated based on the global ML model, wherein the local ML model includes weights with values corresponding to one or more of the plurality of parameters; constructing, for each weight value in each of the received local ML models, a probability distribution for each of the plurality of parameters with corresponding received weight values; sampling, using the constructed probability distribution for each weight value for each of the plurality of parameters, for all of the plurality of local computing devices to generate representative values for each weight; and updating the global ML model by averaging the representative values for each weight for each of the plurality of parameters.