Federated Learning Local Model Regularization for Heterogeneous Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning in heterogeneous environments faces performance degradation due to differences between local and global neural network models, leading to issues like client-drift and catastrophic forgetting, as the training data distributions vary significantly among participants.
Innovation Solution
A method is introduced to calculate the difference between global and local neural network models, determining an additional regularization based on this difference, and incorporating it into the loss function to align the local neural network model with the global model, thereby reducing performance degradation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If federated learning is performed in a heterogeneous environment with significantly different data distribution among participants, then each participant can train using local data without exposing it to the server, but the learning performance deteriorates due to excessive optimization differences between local and global neural networks
Solution Approach 1:
The patent modifies the loss function parameters by adding a regularization term that dynamically adjusts based on the difference between global and local model outputs. This parameter change constrains the local model's optimization direction to prevent excessive deviation from the global model, thereby resolving the performance deterioration caused by heterogeneous data distributions
Solution Approach 2:
The patent implements a feedback mechanism where the difference between global and local model outputs is calculated and used to adjust the regularization strength. This closed-loop feedback controls the divergence between local and global models, ensuring that local training adapts to maintain consistency with the global model while still leveraging local data characteristics
2Loss of information
If local neural network models are trained independently with significant differences in training data distribution, then data privacy is protected, but client-drift and catastrophic forgetting occur causing excessive optimization differences
Solution Approach 1:
The patent applies preliminary anti-action by introducing a regularization term before local training that pre-establishes constraints based on the global model's characteristics. This preventive measure counteracts the tendency of local models to drift away from the global model, addressing client-drift and catastrophic forgetting before they significantly impact performance
Solution Approach 2:
The patent changes the loss function parameters by incorporating a regularization component that dynamically adjusts based on model output differences. This parameter modification maintains model consistency while preserving data privacy, as the constraint is applied through parameter adjustment rather than data sharing
Data Source
AI summary
The present disclosure relates to training a local neural network model based on federated learning in consideration of a heterogeneous environment in which training data are different from each other. An exemplary embodiment of the present disclosure provides a method of training a local neural network model based on federated learning, the method being performed by at least one computing device, the method including: calculating a difference between a global neural network model and a local neural network model; determining an additional regularization for training the local neural network model based on the calculated difference; and training the local neural network model based on a loss function including the determined additional regularization.