Batch Rebalance in Distributed DNN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed Deep Neural Network (DNN) training, the straggler effect occurs due to variable-sized datasets, leading to inefficiencies in computation variance among workers, which impacts scaling efficiency and training time, with existing solutions either affecting test accuracy or wasting computational resources.
Innovation Solution
The proposed solution involves sorting samples by volume and assigning them to local batches using strategies like zigzag round robin balancing or greedy bag balancing, ensuring each worker completes iterations in approximately similar duration without altering the underlying math or convergence, thus minimizing the straggler effect.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If samples are assigned to local batches without sorting by volume, then the assignment process is simple, but computation variance among workers increases due to the straggler effect
Solution Approach 1:
The patent applies preliminary action by sorting samples by volume before assigning them to local batches. This pre-processing step ensures that samples are ordered appropriately before distribution, allowing the system to balance workload across workers and minimize the straggler effect without affecting test accuracy or wasting computational resources.
2Loss of time
If existing solutions are used to address the straggler effect, then computation variance may be reduced, but test accuracy is affected or computational resources are wasted
Solution Approach 1:
The patent applies local quality by making the sample assignment process adaptive to local characteristics of the data. Specifically, it sorts samples based on their volume and assigns them to local batches in a way that balances the computational load across different workers, thereby reducing computation time variance without compromising test accuracy.
3Productivity
If samples are sorted and assigned using zigzag round robin or greedy bag balancing, then workload distribution among workers is balanced, but the assignment complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the sorted samples into multiple local batches and distributing them to different workers using systematic approaches like zigzag round robin or greedy bag balancing. This segmentation strategy balances the workload across workers while maintaining manageable assignment complexity through structured distribution patterns.
Data Source
AI summary
Provided herein are apparatus and methods for batch rebalance in distributed data parallel DNN training. An apparatus includes interface circuitry; and processor circuitry coupled with the interface circuitry, wherein the processor circuitry is to: obtain sorted samples of a mini batch via the interface circuitry, wherein the sorted samples are in an ascend or descend order based on a volume of each of the samples; and assign the sorted samples to each of a plurality of local batches one by one in an order from a first local batch to a last local batch of the plurality of local batches and then from the last local batch to the first local batch until all of the sorted samples are assigned. Other embodiments may also be disclosed and claimed.