Hierarchical ADPSGD Training with Super Learner Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Asynchronous decentralized distributed deep learning training faces challenges with convergence decay as the number of learners increases, leading to inefficiencies in model accuracy and system runtime due to communication bottlenecks and reduced scalability.
Innovation Solution
A hierarchical system employing asynchronous decentralized parallel stochastic gradient descent (ADPSGD) with a super learner manager that coordinates multiple homogenous learners on compute nodes, using quantization and randomized communication patterns to improve training efficiency and reduce network bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of learners is increased to improve training throughput, then productivity is improved, but convergence decay occurs leading to reduced model accuracy
Solution Approach 1:
The system segments learners into hierarchical groups with local aggregators that collect gradients from individual learners and perform local averaging. This segmentation allows the system to scale to more learners while maintaining convergence through localized coordination, preventing the convergence decay that occurs in fully decentralized systems as learner count increases.
Solution Approach 2:
Local aggregators serve as intermediary nodes between individual learners and the global system. These aggregators collect gradients from multiple learners, perform local averaging, and forward aggregated gradients to super learners. This intermediary layer reduces communication overhead and maintains convergence properties even as the total number of learners increases.
2Device complexity
If decentralized communication is used to reduce coordination overhead, then device complexity is reduced, but communication bottlenecks increase reducing productivity
Solution Approach 1:
Communication is segmented into local and global phases. Locally, learners communicate only with their assigned aggregator, reducing communication complexity. Globally, super learners communicate with a limited set of other super learners rather than all learners directly. This segmentation reduces coordination overhead while maintaining training efficiency.
Solution Approach 2:
The system introduces a hierarchical dimension to the communication topology, organizing learners into groups with aggregators and super learners. This hierarchical structure transforms the flat all-to-all communication pattern into a multi-level structure, reducing the communication burden on individual nodes while maintaining overall system productivity.
3Reliability
If synchronous communication is used to ensure coordination, then reliability is improved, but loss of time increases due to waiting for slowest learner
Solution Approach 1:
Synchronization is segmented into local and global levels. Locally, aggregators wait for learners within a group, reducing the waiting time compared to waiting for all learners globally. The hierarchical structure allows parallel progression of different learner groups, reducing overall synchronization waiting time while maintaining coordination reliability.
Solution Approach 2:
The system dynamically adjusts synchronization points at different hierarchical levels. Rather than requiring all learners to synchronize at every global step, the system allows local synchronization at aggregator levels, enabling faster progression. This dynamic approach maintains reliability where needed while reducing time loss through asynchronous progression elsewhere.
Data Source
AI summary
Embodiments of a method are disclosed. The method includes performing a batch of decentralized deep learning training for a machine learning model in coordination with multiple local homogenous learners on a deep learning training compute node, and in coordination with multiple super learners on corresponding deep learning training compute nodes. The method also includes exchanging communications with the super learners in accordance with an asynchronous decentralized parallel stochastic gradient descent (ADPSGD) protocol. The communications are associated with the batch of deep learning training.