Federated Learning Model Training via Quasi-Newton Gradient Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning models face slow convergence speed due to high computational complexity in existing methods like stochastic gradient descent, Newton, and quasi-Newton approaches, particularly in longitudinal federated learning scenarios.
Innovation Solution
A method involving participant devices performing joint encryption training to obtain gradient information, followed by interactive calculations to determine a gradient search direction as a quasi-Newton condition, allowing model parameter updates without calculating the inverse matrix of the Hessian matrix, thereby reducing calculations and communications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If stochastic gradient descent method is used for federated learning model training, then the method is simple to implement, but the convergence speed is slow
Solution Approach 1:
The patent changes the optimization parameters by introducing a quasi-Newton condition that uses gradient information variations to approximate the Hessian matrix inverse. This allows the system to achieve faster convergence comparable to Newton methods while maintaining computational feasibility through parameter approximation rather than exact calculation.
Solution Approach 2:
The patent introduces an intermediary mechanism by using gradient information variations as a mediator to indirectly obtain the search direction without directly calculating the Hessian matrix. This intermediary approach avoids the high computational complexity of direct Hessian calculation while achieving better convergence than standard gradient descent.
2Speed
If Newton approach or quasi-Newton approach is used for federated learning model training, then the convergence speed is improved, but the computational complexity of second derivative is high
Solution Approach 1:
The patent extracts only the essential information needed for optimization by using gradient variations to infer search directions without extracting or calculating the full Hessian matrix. This extraction approach retains the beneficial convergence properties of second-order methods while eliminating the computational burden of calculating second derivatives.
Solution Approach 2:
The patent uses disposable, computationally inexpensive gradient variation calculations instead of expensive Hessian matrix computations. Each gradient variation is a lightweight calculation that can be discarded after use, replacing the heavy computational object of the Hessian matrix while achieving similar optimization effects.
3Ease of operation
If traditional federated learning methods are used, then the training process is straightforward, but the number of calculations and communications is large
Solution Approach 1:
The patent performs partial action by calculating only the necessary gradient variations needed to determine search directions, rather than performing complete Hessian matrix calculations. This partial computation approach reduces the number of communication rounds and calculations while still achieving effective model convergence.
Data Source
AI summary
The disclosure provides a method, an electronic device, and a storage medium for training a federated learning model. According to the method, the electronic device, and the storage medium for training a federated learning model provided in the disclosure, after each of participant devices obtains gradient information of the participant device by performing joint encryption training with a rest of the participant devices, the participant device performs joint training with the rest of the participant devices based on a model parameter variation and a gradient information variation, to obtain a corresponding gradient search direction; then a target participant device of the participant device calculates step information based on the gradient search direction and a model loss function; finally, each of the participant devices updates a model parameter of the participant device based on the gradient search direction and the step information.