Threshold-Based Parameter Update Filtering for Distributed Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed training of large models across multiple computing nodes is resource-intensive and faces challenges such as network latency and bandwidth issues due to the need for continuous synchronization of millions of parameters, especially in interconnected neural networks.
Innovation Solution
Implementing a method where only updates exceeding a threshold are applied and transmitted across nodes, using residual gradients to store smaller updates, and quantizing update values to reduce data volume, thereby optimizing bandwidth usage and maintaining model synchronization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If continuous synchronization of all model parameters is performed across multiple computing nodes, then model consistency is maintained, but network bandwidth consumption increases and training efficiency decreases
Solution Approach 1:
The patent extracts only the essential synchronization information (parameter updates exceeding a threshold) from the complete set of model parameters. Instead of transmitting all parameter values, only those with significant changes are identified and transmitted, reducing network bandwidth consumption while maintaining model consistency across distributed nodes.
Solution Approach 2:
The patent applies partial synchronization by updating only a subset of parameters that exceed a predetermined threshold, rather than synchronizing all parameters. This partial action approach reduces the volume of data transmitted across the network while still maintaining sufficient model consistency for effective distributed training.
2Reliability
If all parameter updates are transmitted and applied across computing nodes, then model synchronization is maintained, but training time increases due to network latency
Solution Approach 1:
The patent extracts only the critical update information that exceeds a threshold from the complete set of parameter changes. By transmitting only these essential updates rather than all parameter changes, the network transmission time is reduced, thereby decreasing training time while maintaining adequate model synchronization.
Solution Approach 2:
The patent implements periodic threshold-based synchronization where updates are transmitted and applied in discrete batches based on whether they exceed a predetermined threshold. This periodic action approach allows computing nodes to process local updates without waiting for continuous synchronization, reducing the impact of network latency on overall training time.
3Loss of energy
If threshold-based filtering of updates is implemented, then bandwidth usage is reduced, but model convergence precision may be affected
Solution Approach 1:
The patent changes the parameter transmission criterion from transmitting all parameter updates to transmitting only those updates that exceed a predetermined threshold. This parameter change in the synchronization strategy reduces bandwidth usage by filtering out minor updates, while the threshold is calibrated to preserve sufficient precision for model convergence.
Data Source
AI summary
Features are disclosed for distributing the training of models over multiple computing nodes (e.g., servers or other computing devices). Each computing device may include a separate copy of the model to be trained, and a subset of the training data to be used. A computing device may determine updates for parameters of the model based on processing of a portion of the training data. A portion of those updates may be selected for application to the model and synchronization with other computing devices. In some embodiments, the portion of the updates is selected based on a threshold value. Other computing devices can apply the received portion of the updates such that the copy of the model being trained in each individual computing device may be substantially synchronized, even though each computing device may be using a different subset of training data to train the model.