Federated Learning Loss Alignment Against Global Model Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning systems face issues with data imbalance among client devices leading to a loss of learning direction and catastrophic forgetting in the global model due to individually trained parameters without reference to a unified direction.
Innovation Solution
A federated learning system that incorporates a loss function for the difference between the predicted values of the global and local models, using a weighted combination of cross-entropy functions to align local models with the global model's learning direction, thereby preventing forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If federated learning is used to avoid data centralization and reduce privacy risks, then data privacy protection is improved, but the learning direction is lost causing the forgetting problem
Solution Approach 1:
The patent introduces a feedback mechanism where the global model's predicted values are fed back to client devices. These predicted values serve as feedback signals that guide local model training, ensuring local models do not deviate from the global learning direction while maintaining data privacy through federated learning architecture.
Solution Approach 2:
The global model is trained first to establish the correct learning direction and produce predicted values before local model training begins. This preliminary action ensures that local models have a reference framework to follow, preventing the forgetting problem from the outset while maintaining the privacy benefits of federated learning.
2Productivity
If local models are trained independently using only local data, then training speed is improved, but parameter updates deviate from global model direction
Solution Approach 1:
The global model's predicted values act as an intermediary between the global model and local models. This intermediary provides guidance signals that mediate the training process, allowing local models to train independently at high speed while still receiving directional guidance to maintain parameter update accuracy aligned with the global model.
3Productivity
If data is centralized in cloud memory for model learning, then learning efficiency is improved, but network costs and data privacy risks increase
Solution Approach 1:
The patent segments the model training process into global model training at the central server and local model training at client devices. This segmentation allows efficient parallel training without centralizing data, reducing network transfer costs and eliminating the need to move large datasets between cloud and clients while maintaining high learning efficiency.
Data Source
AI summary
Proposed is a federated learning system. The federated learning system comprises: a central server configured to transmit at least one global parameter of a global model to each client device, receive at least one local parameter of a local model trained from each of client devices, and update the global model using the at least one local parameter; and a plurality of client devices configured to train the local model by applying a loss between a predicted value of the global model and a predicted value of the local model possessed by itself to a loss function, and transmit at least one local parameter of the trained local model to the central server.