Multi-GPU Neural Network Training via Stratified Data Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training neural networks on multiple GPUs in parallel is inefficient due to varying processing times for data samples of different sizes, leading to wasted computational resources as some GPUs wait for others to finish.
Innovation Solution
By sampling data samples of similar sizes across GPUs, ensuring that each GPU processes data samples with similar processing times, thereby minimizing jitter and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple GPUs process data samples of different sizes in parallel, then the training can utilize more computational resources, but some GPUs will wait for others to finish causing wasted computational resources
Solution Approach 1:
The patent changes the parameter of data sample size distribution by stratifying the dataset into size buckets and assigning samples to GPUs based on their size characteristics. This ensures that GPUs receive data samples of similar sizes, balancing the computational load and eliminating wait-time waste while maintaining parallel training efficiency.
2Adaptability or versatility
If multiple GPUs process data samples of different sizes in parallel, then more diverse data can be processed, but processing time varies causing jitter and reduced efficiency
Solution Approach 1:
The patent introduces a new parameter dimension for data assignment by creating size-based buckets and using stratified sampling. This transforms the data assignment process from random distribution to controlled distribution based on sample size, thereby reducing processing time variation while preserving data diversity through systematic sampling across different size categories.
Solution Approach 2:
The patent performs preliminary classification of data samples into size buckets before the actual training process. This pre-organization of data by size ensures that when data is distributed to GPUs, the processing times are already balanced, eliminating jitter during parallel execution.
3Device complexity
If a single GPU performs all computations, then the system is simple to manage, but the training process becomes slow and inefficient
Solution Approach 1:
The patent segments the training workload across multiple GPUs by dividing the dataset into different size buckets and distributing samples to multiple GPU devices. This segmentation enables parallel processing, significantly improving training speed while maintaining manageable system complexity through structured data distribution protocols.
Data Source
AI summary
Apparatuses, systems, and techniques to train a neural network on multiple graphics processing units (GPUs). In at least one embodiment, the neural network may be trained in parallel based, at least in part, on two or more randomly selected, similarly-sized portions of one or more datasets.