moDNN Framework for GPU Memory Optimization in DNN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face memory constraints during training on graphics processing units (GPUs), as the memory usage of state-of-the-art DNNs exceeds the capacity of current high-end GPUs, leading to accuracy loss when scaling down the network to fit within the memory limits.
Innovation Solution
The proposed memory optimal DNN training framework, moDNN, employs data offloading and prefetching, automatic sub-batch size selection, and convolution process optimization to reduce memory usage while maintaining accuracy, enabling the training of larger-scale DNNs on single or multiple GPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the batch size is reduced to fit the DNN training data into GPU memory, then the memory usage is reduced, but the training accuracy deteriorates
Solution Approach 1:
The training batch is divided into multiple sub-batches that can be processed independently. Each sub-batch fits within the GPU memory constraints, allowing the system to process larger overall batches by accumulating gradients across multiple smaller processing passes. This segmentation resolves the contradiction by enabling large effective batch sizes (for accuracy) while using small actual batch sizes per GPU pass (for memory efficiency).
Solution Approach 2:
The system performs preliminary calculations to determine optimal sub-batch sizes and numbers of passes required to achieve the target effective batch size. This preliminary planning allows the system to pre-compute the training strategy that will maintain accuracy while fitting memory constraints, resolving the contradiction before training begins.
2Quantity of substance
If the network scale is reduced to fit within GPU memory, then the memory usage is reduced, but the model accuracy deteriorates
Solution Approach 1:
The system segments the training process into multiple passes over the same large-scale network, rather than reducing the network size. Each pass processes a portion of the batch, and gradients are accumulated across passes. This allows the full-scale network (for accuracy) to be used while the per-pass memory usage remains within GPU constraints.
3Quantity of substance
If out-of-core training is used to reduce memory usage, then the memory usage is reduced, but the training time increases
Solution Approach 1:
The system maintains continuous GPU utilization by overlapping data transfer operations with computation operations. While the GPU processes one sub-batch, data for the next sub-batch is being transferred to the GPU memory simultaneously. This continuity minimizes idle time and resolves the contradiction by keeping the system productive throughout the out-of-core training process.
Solution Approach 2:
The system performs preliminary data prefetching and transfer operations before they are strictly needed, preparing data in advance during periods when the GPU is occupied with other computations. This preliminary action reduces waiting time during the training process, maintaining productivity while enabling out-of-core training.
Data Source
AI summary
A method operating a Graphics Processing Unit (GPU) memory can be provided by accessing specified training parameters used to train a Deep Neural Network (DNN) using a GPU with a local GPU memory, the specified training parameters including at least a specified batch size of samples configured to train the DNN. A sub-batch size of the samples can be defined that is less than or equal to the specified batch size of samples in response to determining that an available size of the local GPU memory is insufficient to store all data associated with training the DNN using one batch of the samples. Instructions configured to train the DNN using the sub-batch size can be defined so that an accuracy of the DNN trained using the sub-batch size is about equal to an accuracy of the DNN trained using the specified batch size of the samples.