Offloading Neural Network Activations to CPU Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks face memory constraints on GPUs, limiting the amount of training data that can be processed, making certain tasks untrainable due to high memory requirements during the training process.
Innovation Solution
The technique involves selectively offloading computed activations from a GPU to a CPU during forward propagation, freeing up GPU memory, and transferring them back during backward propagation as needed, allowing for larger datasets to be trained by reducing memory requirements on the GPU.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If activations are stored on GPU memory during forward propagation, then gradient computation can be performed, but GPU memory capacity is exceeded for large datasets
Solution Approach 1:
The patent introduces a new spatial dimension for storage by moving activations from GPU memory to CPU memory, effectively expanding the available storage capacity beyond the limited GPU memory space. This dimensional transition allows handling of larger datasets that would otherwise exceed GPU memory capacity.
Solution Approach 2:
The patent introduces CPU memory as an intermediary storage layer between GPU computation and persistent storage. Activations are temporarily stored in CPU memory during training, allowing the GPU to process larger batches without being constrained by its limited memory capacity.
2Speed
If all activations are kept on GPU, then gradient computation is fast, but memory requirements exceed available GPU memory
Solution Approach 1:
The patent segments the storage location of activations into two parts: frequently accessed activations remain in GPU memory for fast gradient computation, while less frequently accessed activations are stored in CPU memory. This segmentation allows the system to balance between computation speed and memory capacity.
Solution Approach 2:
The patent changes the storage location parameter of activations from exclusively GPU memory to a distributed configuration involving both GPU memory and CPU memory. This parameter change enables the system to handle larger datasets while maintaining acceptable gradient computation performance.
3Productivity
If GPU memory is allocated for training data, then training can proceed, but the amount of trainable data is limited by memory capacity
Solution Approach 1:
The patent makes the activation storage system universal by allowing activations to be stored in either GPU memory or CPU memory depending on their access patterns and the available resources. This multi-functionality enables the training system to adapt to different dataset sizes and GPU memory capacities, maximizing training throughput across various scenarios.
Data Source
AI summary
Techniques for selectively offloading data that is computed by a first processing unit during training of an artificial neural network onto memory associated with a second processing unit and transferring the data back to the first processing unit when the data is needed for further processing are described herein. For example, the first processing unit may compute activations for operations associated with forward propagation. During the forward propagation, one or more of the activations may be transferred to a second processing unit for storage. Then, during backpropagation for the artificial neural network, the activations may be transferred back to the first processing unit as needed to compute gradients.