Federated Learning With Variance-Reduced Gradients for Faster Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep neural networks is inefficient and challenging due to the large number of component functions, leading to suboptimal gradient descent methods that require excessive computational resources and expose data redundantly, hindering convergence and privacy.
Innovation Solution
Federated learning with variance-reduced stochastic gradient descent is employed, utilizing an outer and inner loop optimization framework, where gradients are calculated across multiple clients, and the learning rate is dynamically adjusted based on gradient quality, allowing for efficient resource balancing and reduced exposure of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional gradient descent methods are used to train deep neural networks, then the model can be trained using simple algorithms, but the training process requires excessive computational resources and fails to converge efficiently
Solution Approach 1:
The patent segments the training process into multiple rounds where each round processes a subset of component functions. Instead of computing gradients over all component functions in each iteration, the method divides the large-scale optimization problem into smaller sub-problems that can be solved more efficiently with reduced computational resources while maintaining convergence properties.
Solution Approach 2:
The patent dynamically adjusts optimization parameters including the learning rate and the number of component functions processed per round. By changing these parameters adaptively based on progress and resource availability, the method achieves better training efficiency without requiring excessive computational resources, resolving the contradiction between productivity and energy use.
2Loss of information
If data is exposed during training to enable gradient calculation, then the model can learn from the data, but data privacy is compromised
Solution Approach 1:
The patent extracts only the necessary gradient information from the training data without exposing the raw data itself. By computing gradients locally and transmitting only the gradient updates rather than the underlying data, the method preserves data utility for model learning while protecting data privacy, effectively separating the information needed for training from the sensitive data.
3Measurement precision
If the number of training steps is increased to achieve convergence, then the model accuracy improves, but the training time and computational cost increase
Solution Approach 1:
The patent performs preliminary actions by pre-processing component functions and preparing data structures before the main training loop. This preliminary preparation enables faster gradient computations during training, allowing the model to achieve convergence in fewer steps and reducing overall training time while maintaining model accuracy.
Solution Approach 2:
The patent maintains continuous useful action by overlapping computation and communication operations, and by keeping the optimization process continuously active rather than stopping and starting. This continuity reduces idle time and allows the model to progress more efficiently toward convergence, improving the trade-off between accuracy and training time.
Data Source
AI summary
An approach to federated learning of a machine learning model may be provided. The approach may include broadcasting hyperparameters of a machine learning model to one or more client computing devices from a primary device associated with an outer loop or an inner loop. A gradient for the loss function may be calculated at the client device if previous gradients have been sufficiently large. If gradients exceeds a threshold, the client can send the mini-batch of gradients or the difference of the mini-batch of gradients back to the primary device. A search direction may be calculated based on the full gradient of the loss function for an outer loop or the mini-batch of gradient differences for an inner loop. A learning rate step may be calculated from the search direction. The hyperparameter may be updated for the inner loop based on the learning rate.