Data Loading for Distributed Deep Learning Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed deep learning training, synchronization of weight updates across multiple GPUs is inefficient due to uneven data processing, leading to increased waiting times and communication overhead.
Innovation Solution
The training data set is divided into subsets based on data file sizes, with data files loaded and reallocated among processors in a manner that balances the batch size and file proportions, and sorted for sequential distribution to reduce size deviations and communication costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data files are loaded uniformly to multiple GPUs without reallocation, then the data loading process is simple, but the processing time becomes uneven across GPUs causing synchronization delays
Solution Approach 1:
The system performs preliminary actions by dividing the training data set into subsets based on file sizes before loading, and pre-calculates the optimal distribution plan. This preliminary organization enables balanced allocation across GPUs, ensuring uniform processing times and reducing synchronization waiting time without complicating the actual loading operation.
Solution Approach 2:
The system dynamically reallocates data files from initially loaded GPU memory to achieve balanced processing loads. By monitoring and adjusting the distribution of data files based on actual file sizes and processing requirements, the system adapts to variations in data characteristics, ensuring all GPUs complete their batches simultaneously and minimizing synchronization delays.
2Productivity
If data files are reallocated among processors to balance processing times, then synchronization efficiency improves, but communication overhead increases
Solution Approach 1:
The system segments the training data set into multiple subsets based on file size characteristics before distribution. By organizing data into size-based subsets and allocating them strategically across GPUs, the system achieves balanced processing loads that improve synchronization efficiency while minimizing the need for frequent reallocation communications, thus reducing communication overhead.
3Adaptability or versatility
If data files of varying sizes are processed by multiple GPUs, then data diversity is maintained, but processing time uniformity deteriorates
Solution Approach 1:
The system applies local quality by assigning specific size ranges of data files to specific GPUs based on their capabilities and current load. By dividing the training data into subsets with controlled size variations and strategically allocating them to different GPUs, the system maintains data diversity while ensuring each GPU processes files within a manageable size range, achieving uniform processing times across all processors.
Data Source
AI summary
A processor-implemented method with data loading includes: dividing a training data set into a plurality of subsets based on sizes of a plurality of data files included in the training data set; loading, from each of the plurality of subsets, a portion of data files in the subset to a plurality of processors based on a proportion of a number of data files of the plurality of subsets in the subset and a batch size of distributed training; and reallocating, based on sizes of data files loaded to processors in a same group among the plurality of processors, the loaded data files to the processors in the same group.