Hierarchical Aggregation for Distributed Deep Learning Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training machine learning models is time-consuming due to the need for large datasets, and distributed learning approaches face challenges in efficiently synchronizing models across parallel processors, with synchronization costs heavily dependent on network topology and interconnect speeds.
Innovation Solution
A hierarchical grouped aggregation system is implemented, where lower layers with faster interconnects perform frequent synchronizations within smaller processor groups to facilitate model convergence, while upper layers with slower interconnects perform less frequent updates across larger sets, optimizing synchronization strategies based on processor topology to train models faster with a larger number of processors in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed learning is used to train models faster with multiple processors, then training speed is improved, but synchronization difficulty increases
Solution Approach 1:
The patent divides the distributed processing system into a hierarchical structure with multiple layers. Each layer manages a subset of processors and performs synchronization locally before propagating updates to higher layers. This segmentation reduces the complexity of global synchronization by breaking it down into smaller, manageable local synchronization tasks within each layer.
Solution Approach 2:
The patent introduces a hierarchical dimension to the synchronization process, organizing processors into multiple layers rather than a flat structure. This dimensional change allows synchronization to occur at different levels of the hierarchy, with lower layers handling fine-grained updates and upper layers handling coarser-grained updates, thereby reducing overall synchronization complexity.
2Reliability
If frequent synchronization is performed across all processors, then model convergence is improved, but communication overhead increases
Solution Approach 1:
The patent segments the synchronization process into multiple hierarchical layers, where each layer performs synchronization at different frequencies. Lower layers perform more frequent synchronization for their local processor groups, while upper layers perform less frequent synchronization. This segmentation allows the system to maintain model convergence while reducing overall communication overhead by avoiding full-system synchronization at every step.
Solution Approach 2:
The patent applies partial synchronization by having only subsets of processors synchronize at any given time, rather than requiring all processors to synchronize simultaneously. This partial action approach maintains sufficient model convergence while significantly reducing communication overhead compared to full-system synchronization.
3Productivity
If more processors are used in parallel, then training capacity is improved, but synchronization cost increases
Solution Approach 1:
The patent organizes processors into a hierarchical structure with multiple layers, where each layer manages a specific subset of processors. This segmentation allows the system to scale to more processors while controlling synchronization costs, as synchronization occurs locally within each layer before propagating to higher layers, rather than requiring all processors to synchronize directly with each other.
Solution Approach 2:
The patent introduces a hierarchical dimension to processor organization, arranging processors in multiple layers rather than a flat structure. This dimensional change enables the system to handle more processors efficiently by reducing the synchronization cost through the hierarchical structure, where synchronization propagates through layers rather than requiring direct communication between all processor pairs.
Data Source
AI summary
A computer implemented method trains distributed sets of machine learning models by training each of the distributed machine learning models on different subsets of a set of training data, performing a first layer model synchronization operation in a first layer for each set of machine learning models, wherein each model synchronization operation in the first layer generates first updates for each of the machine learning models in each respective set, updating the machine learning models based on the first updates, performing a second layer model synchronization operation in a second layer for first supersets of the machine learning models wherein each model synchronization in the second layer generates second updates for updating each of the machine learning models in the first supersets based on the second updates such that each machine learning model in a respective first superset is the same.