Entropy-Based Client Selection in Federated Learning Under Data Drift
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning systems face challenges with slow convergence and biased results due to data heterogeneity and concept drift, particularly in environments with non-IID data, leading to increased communication overhead and inefficient client selection.
Innovation Solution
Entropy-based client selection mechanism that determines prediction entropy from model confidences to identify and exclude drifted and noisy clients, clustering clients based on entropy values to improve convergence and handle heterogeneous data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If random client selection is used in federated learning, then the implementation is simple, but the convergence speed decreases and communication overhead increases
Solution Approach 1:
The patent changes the selection parameter from random to entropy-based scoring. Each client is assigned an entropy value that quantifies their data heterogeneity and drift characteristics. Clients are then selected based on their entropy scores, allowing the system to adaptively choose participants that optimize convergence while managing communication overhead through targeted selection rather than universal participation.
2Productivity
If influence-based client selection is used to handle data heterogeneity, then the convergence speed improves, but computational and communication overhead increases significantly
Solution Approach 1:
The patent extracts the essential selection criterion from complex influence-based methods. Instead of computing full Hessian Vector Products or using randomized Kaczmarz methods that require extensive communication, the system extracts and uses only entropy values as the selection metric. These entropy values can be computed locally with minimal communication, capturing the essential information needed to identify drifted and noisy clients without the computational burden of influence-based approaches.
3Device complexity
If loss-based statistical model utility is used to quantify client importance, then the communication overhead is reduced, but the approach is limited in resource-constrained edge use cases
Solution Approach 1:
The patent employs cheap, lightweight entropy calculations that can be performed efficiently on resource-constrained edge devices. Rather than using computationally intensive influence-based methods or complex loss aggregations, the system uses entropy values that capture client characteristics with minimal computational resources. This allows the system to operate effectively on mobile devices and edge computing platforms while maintaining low communication overhead.
Data Source
AI summary
Entropy based federated learning is disclosed. In federated learning, a model is trained at multiple clients using corresponding local data. An entropy associated with the local training is determined and provided, along with a model update, to a central server. The central server selects specific clients to participate in the current aggregation operation based on the entropy values. This minimizes the number of drifted and noisy clients that are included in the aggregation operation. The model updates of the selected clients are aggregated and a new or updated global model is generated. To aid in accounting for data heterogeneity, the clients may be grouped and a new or updated global model may be generated for each of the groups using model updates from corresponding selected clients.