Neural Network Memory Management via Tensor Timing Slack
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face memory constraints due to limited GPU memory, limiting the depth of the network and size of data samples, which affects accuracy, and existing methods struggle to efficiently manage memory usage during computation.
Innovation Solution
The method involves computing tensor timing slacks, filtering candidate tensors, and inserting memory reduction and restoration operations into the data flow graph to reduce GPU memory usage by transferring data to CPU memory or compressing data, allowing for larger data samples to be processed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPU memory capacity is increased to handle larger data samples and deeper networks, then processing capability and accuracy improve, but hardware cost and complexity increase
Solution Approach 1:
The patent introduces CPU memory as an intermediary storage layer between GPU computation units and persistent storage. Data is transferred from CPU memory to GPU memory on-demand, allowing the system to process datasets larger than GPU memory capacity while maintaining efficient computation. This mediator approach enables handling of ImageNet-scale datasets without requiring proportionally large GPU memory investments.
Solution Approach 2:
The patent segments the data processing workflow into distinct phases: data loading from storage to CPU memory, selective transfer of required data subsets to GPU memory, computation execution, and result retrieval. This segmentation allows the system to process large datasets incrementally, loading only the necessary portions into GPU memory at any given time, thereby reducing peak memory requirements while maintaining processing capability.
2Quantity of substance
If data is transferred from GPU memory to CPU memory to free up space, then GPU memory availability increases, but data access time increases
Solution Approach 1:
The patent implements preliminary data loading from persistent storage to CPU memory before GPU processing begins. This advance preparation ensures that data is readily available in system memory when needed, reducing the frequency and impact of transfers between GPU and CPU memory during computation. The system proactively manages data movement to minimize computational interruptions.
Solution Approach 2:
The patent maintains continuous data flow between CPU and GPU memory through optimized transfer protocols and buffering mechanisms. By keeping data in CPU memory during GPU computation and enabling rapid bidirectional transfer, the system ensures that computational actions continue without significant interruptions, minimizing the time loss associated with memory transfers while maintaining adequate GPU memory availability.
Data Source
AI summary
Processing a neural network data flow graph having a set of nodes and a set of edges. An insertion point is determined for a memory reduction or memory restoration operation. The determination is based on computing tensor timing slacks (TTS) for a set of input tensors; compiling a candidate list (SI) of input tensors, from the set of input tensors, using input tensors having corresponding TTS values larger than a threshold value (thTTS); filtering the SI to retain input tensors whose size meets a threshold value (thS); and determining an insertion point for the operation using the SI based on the filtering. A new data flow graph is generated or an existing one is modified using this process.