Decentralized Learning Activation Function for Faster Model Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing federated learning methods face challenges in achieving model fitness at the local phase, leading to under-fitting or over-fitting, which affects the convergence rate and communication overhead, especially in scenarios with limited data or non-representative data distributions.
Innovation Solution
Implementing an activation function that preserves agreements and discourages disagreements between local and reference parameter sets during the training of local ML models, using a contrastive learning approach to reduce the number of communication rounds required for convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If agents learn under-fitted models to their local data, then the global model needs many rounds to converge to the desired solution, but this increases communication overhead and time cost
Solution Approach 1:
The patent applies preliminary action by performing local model training with contrastive regularization before global aggregation. The contrastive loss function is computed in advance during local training to establish meaningful parameter relationships, so that when models are aggregated globally, convergence occurs faster with fewer communication rounds.
Solution Approach 2:
The patent implements feedback through the contrastive loss mechanism that continuously monitors and adjusts local model parameters during training. The loss function provides real-time feedback on parameter agreements and disagreements, guiding the optimization process to achieve better model fitness before communication occurs.
2Reliability
If agents learn over-fitted models to their local data, then the global model may diverge or converge to a poor solution, but achieving good local fit requires more communication rounds
Solution Approach 1:
The contrastive loss function provides continuous feedback during local training by measuring parameter agreements and disagreements. This feedback mechanism prevents over-fitting by penalizing excessive deviation from reference parameters while still allowing meaningful local adaptation, thus maintaining both model fitness and convergence rate.
Solution Approach 2:
The patent changes parameters by introducing contrastive regularization terms that modify the optimization landscape. The contrastive loss dynamically adjusts parameter updates based on agreement metrics, transforming the training process to balance local fit quality with global convergence properties.
3Loss of time
If the degree of model fitness is not properly controlled, then communication overhead increases, but controlling it requires complex mechanisms
Solution Approach 1:
The patent simplifies the training mechanism by changing parameters to use contrastive loss with agreement metrics. This approach controls model fitness through a relatively simple mathematical formulation that directly measures parameter relationships, avoiding the need for complex regularization schedules or multiple control mechanisms.
Data Source
AI summary
A computer-implemented method is provided performed by a client computing device for decentralized learning based on local learning at the client computing device is provided. The method includes training a local M, model based on an activation function using a local parameter set and a reference parameter set to obtain a setting for respective 5 local parameters in the local parameter set that minimizes a training loss wherein the activation function preserves agreements and discourages disagreements between the local parameter set and the reference parameter set. The method further includes sending the trained local ML model to a server computing device. The method further includes receiving, from the server computing device, a global ML model that meets a convergence criterion. A 10 method performed by a server computing device, and related methods and apparatuses are also provided.