Federated Learning with Heterogeneous Labels and New Class Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current federated learning approaches assume uniform model architectures and label distributions across client devices, leading to overfitting or underfitting, and cannot handle unique or heterogeneous labels effectively, resulting in poor global models.
Innovation Solution
A method for federated learning that allows clients to choose their own architectures and handles heterogeneous labels using a zero-shot learning mechanism with anonymized Data Impressions, synthesized from class similarity matrices, and unsupervised clustering to validate new labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If federated learning uses uniform model architectures and label distributions across client devices, then the training process is simplified and coordination is easier, but the model performance deteriorates due to overfitting or underfitting and inability to handle heterogeneous labels
Solution Approach 1:
The patent allows each client device to have its own local model architecture tailored to its specific data characteristics and label distribution, rather than enforcing a uniform architecture. This enables each local model to be optimized for its local conditions while still participating in federated learning, thereby improving overall model performance without significantly complicating the coordination process
Solution Approach 2:
The patent introduces parameters to track and manage label distributions across different client devices, allowing the system to adapt to heterogeneous labels by changing the representation and handling of label information. This enables the federated learning process to accommodate diverse label distributions while maintaining coordination through parameter-based management rather than structural complexity
2Adaptability or versatility
If federated learning allows clients to choose their own model architectures to handle heterogeneous labels, then the model adaptability and performance on local data improve, but the system complexity increases due to diverse model structures and label distributions
Solution Approach 1:
The patent creates a universal federated learning framework that can handle multiple model architectures and label distributions through a common interface and coordination mechanism. The server maintains a unified view of the learning process while allowing diverse local models to participate, achieving multi-functionality that accommodates heterogeneous architectures without proportionally increasing system complexity
Solution Approach 2:
The patent introduces an intermediary layer at the server that manages the diversity of local models through parameter-based coordination. This intermediary handles label distribution tracking and model aggregation, acting as a mediator that simplifies the interaction between diverse local models and the central coordination system, thereby limiting the growth of overall system complexity
3Quantity of substance
If federated learning aggregates updates from all clients uniformly, then the global model benefits from diverse data sources, but the label distribution heterogeneity causes inaccurate global label assignments and reduced accuracy
Solution Approach 1:
The patent implements feedback mechanisms where the server tracks label distributions from each client and uses this information to adjust the aggregation process. The label distribution parameters are fed back into the model update aggregation, allowing the system to account for heterogeneity when combining updates, thereby maintaining accurate global label assignments while benefiting from diverse data sources
Solution Approach 2:
The patent applies partial weighting to client updates based on their label distribution characteristics rather than uniform aggregation. By selectively emphasizing or de-emphasizing certain client contributions based on label overlap and distribution similarity, the system achieves more accurate global label assignments while still incorporating diversity from multiple data sources
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A method for distributed machine learning (ML) which includes providing a first dataset including a first set of labels to a plurality of local computing devices including a first local computing device and a second local computing device. The method further includes receiving, from the first local computing device, a first set of ML model probabilities values from training a first local ML model using the first set of labels. The method further includes receiving, from the second local computing device, a second set of ML model probabilities values from training a second local ML model using the first set of labels and one or more labels different from any label in the first set of labels. The method further includes generating a weights matrix using the received first set of ML model probabilities values and the received second set of ML model probabilities values. The method further includes generating a third set of ML model probabilities values by sampling using the generated weights matrix.