GPU-CPU Memory Offloading for Deep Learning Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning systems utilizing graphics processing units (GPUs) face limitations due to limited on-board memory, which restricts their ability to accommodate large or complex deep learning networks, leading to performance bottlenecks and inefficiencies.
Innovation Solution
A system comprising a graphics processing unit (GPU) and a central processing unit (CPU) memory, where the GPU processes data to train deep neural networks, with the CPU memory storing and providing data to the GPU via compression schemes and half-precision floating-point formats, facilitating improved performance by reducing computing bottlenecks and increasing batch sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If graphics processing unit is used to execute deep learning networks, then processing capability is improved, but memory capacity is insufficient to accommodate large or complex networks
Solution Approach 1:
The patent introduces host memory as an intermediary storage component that supplements the GPU's limited onboard memory. The host memory stores portions of training data and model parameters, allowing the GPU to access additional memory resources through the memory bus without being constrained by its own limited capacity. This mediator enables the system to accommodate large deep learning networks that would otherwise be too large for the GPU's onboard memory alone.
2Quantity of substance
If data is transferred between graphics processing unit and central processing unit memory, then memory capacity is extended, but data transfer time increases processing duration
Solution Approach 1:
The patent implements preliminary action by pre-loading training data and model parameters into the GPU's onboard memory before the deep learning computation begins. The system transfers data from host memory to GPU memory in advance, so that when the actual computation starts, the necessary data is already available in the GPU's fast memory, minimizing transfer time during the critical computation phase.
Solution Approach 2:
The patent employs periodic action through the use of memory buses that enable intermittent data transfers between host memory and GPU memory during the deep learning process. The system periodically loads new data batches or model parameters from host memory to GPU memory as needed during training, allowing continuous operation while managing memory capacity constraints through scheduled data transfers.
Data Source
AI summary
Techniques that facilitate model support in deep learning are provided. In one example, a system includes a graphics processing unit and a central processing unit memory. The graphics processing unit processes data to train a deep neural network. The central processing unit memory stores a portion of the data to train the deep neural network. The graphics processing unit provides, during a forward pass process of the deep neural network that traverses through a set of layers for the deep neural network from a first layer of the set of layers to a last layer of the set of layers that provides a set of outputs for the deep neural network, input data for a layer from the set of layers for the deep neural network to the central processing unit memory.