Federated Learning Global Embedding Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning methods face challenges when applied to real-world scenarios due to heterogeneous and varying sizes of datasets on edge devices, making it infeasible to train a single global model effectively.
Innovation Solution
A federated learning scheme that trains a global embedding along with local task-specific networks, where each edge device generates a feature vector based on input data and global model parameters, and modifies model parameters to minimize a loss value while rewarding improvements and penalizing deteriorations across tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single global model is trained across heterogeneous edge devices, then data privacy and security are maintained, but model training becomes infeasible due to dataset heterogeneity and size variations
Solution Approach 1:
The patent segments the monolithic global model into a shared global model and multiple task-specific models. Each edge device maintains its own task-specific model while sharing the global model, allowing independent training on local heterogeneous data while preserving data privacy. This segmentation enables feasible model training across diverse datasets.
Solution Approach 2:
The shared global model serves multiple tasks across different edge devices simultaneously. It is trained on aggregated data from all devices and can be applied to various local tasks, providing universal functionality that adapts to heterogeneous data distributions while maintaining data security.
2Measurement precision
If local task-specific models are trained independently on each edge device, then task performance is improved, but the quality of the shared global model deteriorates
Solution Approach 1:
The patent implements feedback mechanisms where task-specific models provide performance information back to the shared global model. The global model is retrained using aggregated data from all edge devices, and updated parameters are distributed back to local devices. This feedback loop ensures that improvements in local task performance do not compromise global model quality.
Solution Approach 2:
The patent merges the training processes of multiple task-specific models into a unified global training process. Data from all edge devices are aggregated (in a privacy-preserving manner) to retrain the shared global model, combining the strengths of local adaptations while maintaining overall model coherence and quality.
3Measurement precision
If the shared global model is updated frequently to improve task performance, then individual task performance improves, but training time and computational resources increase
Solution Approach 1:
The patent employs periodic updates of the shared global model rather than continuous updates. Edge devices perform local training on their task-specific models, and the global model is retrained at scheduled intervals using aggregated data. This periodic approach balances task performance improvement with reasonable training time and resource consumption.
Solution Approach 2:
The patent uses partial updates where only certain parameters of the global model are updated based on aggregated gradients from edge devices, rather than complete retraining. This selective updating reduces computational overhead and training time while still achieving performance improvements.
Data Source
AI summary
The present invention relates to the technical field of federated learning. Subject matter of the present invention is a method for (re-)training a federated learning system, a computer system for carrying out the method, and a non-transitory computer-readable storage medium comprising processor-executable instructions with which to perform an operation for (re-)training a federated learning system.