Distributed Training Data Loading with Size-Based File Packing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for distributed training of deep learning models face inefficiencies due to uneven data processing across GPUs, leading to synchronization overhead and degraded convergence behavior, particularly when data files of varying sizes are not adequately considered in the loading process.
Innovation Solution
A method and apparatus that divide training datasets into sub-sets based on data file sizes, determine packing combinations considering the ratio of data files between sub-sets and batch size, and reallocate packed data files to processors within the same group to optimize data distribution and reduce communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data files are loaded uniformly across GPUs without considering file sizes, then the data loading process is simple, but synchronization overhead increases and convergence behavior degrades due to uneven data processing
Solution Approach 1:
The training dataset is divided into multiple sub-sets based on data file sizes, with each GPU assigned to process specific sub-sets. This segmentation ensures that each GPU handles data files of comparable sizes, balancing the processing workload and reducing synchronization waiting time while maintaining a manageable loading process through systematic organization
Solution Approach 2:
Different GPUs are assigned different sub-sets of data files based on their size characteristics. Each GPU processes data with locally appropriate size properties, ensuring that no single GPU is bottlenecked by excessively large files while others have smaller files, thus optimizing overall synchronization efficiency
2Ease of operation
If data files of varying sizes are not considered in the loading process, then the loading procedure is straightforward, but operational efficiency and convergence are degraded
Solution Approach 1:
Before the actual training process begins, the training dataset is pre-processed and divided into sub-sets based on data file sizes. This preliminary organization ensures that when data is loaded during training, each GPU receives appropriately sized files, optimizing operational efficiency from the start without adding complexity to the loading procedure itself
3Device complexity
If data files are not packed and reallocated based on size ratios, then the data distribution process is simple, but synchronization overhead increases
Solution Approach 1:
The data distribution process incorporates file size as a key parameter for determining packing combinations. By changing the distribution strategy from uniform allocation to size-based allocation, the system optimizes synchronization efficiency. The packing combination is determined by calculating the ratio of data files between sub-sets and the batch size, ensuring balanced workload distribution
Data Source
AI summary
A processor-implemented method with data loading includes: based on sizes of a plurality of data files in a training dataset, dividing the training dataset into a plurality of sub-sets; loading some data files in each sub-set into a plurality of processors; determining a packing combination of one or more data files loaded to processors in a same group among the plurality of processors, based on a ratio of a number of data files between the plurality of sub-sets and a batch size of distributed training; determining packed data files by packing the one or more data files according to the packing combination; and reallocating the packed data files to the processors in the same group.