Computational Graph Rewriting for GPU Memory Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks often exceed the GPU memory capacity, necessitating an approach to manage and optimize memory consumption to support their operation.
Innovation Solution
A method involving the construction of a categorized topological ordering of a computational graph, where nodes are arranged in levels, and the graph is rewritten by linearizing nodes to avoid overlapping memory consumption when it exceeds a threshold, using swap-out and swap-in operations to manage GPU memory effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep neural networks are made deeper and larger to improve model capacity, then model performance is improved, but GPU memory consumption increases beyond available capacity
Solution Approach 1:
The computational graph is divided into multiple levels through topological sorting, where each level contains operations that can be executed independently. This segmentation allows the system to process large neural networks by breaking them into manageable chunks that fit within GPU memory constraints, enabling training of deeper and larger models without exceeding memory capacity.
Solution Approach 2:
The patent introduces a temporal dimension to memory management by implementing swap-out and swap-in operations between GPU and CPU memory. This transforms the static memory allocation problem into a dynamic multi-dimensional memory space, allowing data to be moved between different memory tiers based on when it is needed, thereby supporting larger models than would fit in GPU memory alone.
2Speed
If multiple computational nodes are executed in parallel within the same level, then computation speed is improved, but GPU memory consumption increases due to overlapping memory usage
Solution Approach 1:
The system performs preliminary topological sorting and level assignment before execution, and estimates memory consumption for each level in advance. This preliminary analysis allows the system to identify potential memory conflicts and adjust the execution plan accordingly, enabling parallel execution within levels while preventing memory overflow by proactively managing resource allocation.
Solution Approach 2:
The patent implements dynamic memory management through swap-out and swap-in operations that adapt to actual memory consumption patterns during execution. When memory usage exceeds thresholds, the system dynamically swaps less frequently needed data to CPU memory and brings in required data as needed, allowing flexible parallel execution without fixed memory allocation constraints.
Data Source
AI summary
A computer-implemented method is provided for managing GPU memory consumption by computational graph rewriting. The method includes constructing, by a hardware processor, a categorized topological ordering of a computational graph. The categorized topological ordering includes multiple computational nodes arranged in multiple levels. The method further includes estimating, by the hardware processor, the GPU memory consumption responsive to a level including two or more computational nodes from among the multiple computational nodes. The method also includes rewriting, by the hardware processor, the computational graph by linearizing the two or more computational nodes in the level to avoid overlapping of the GPU memory consumption by the two or more computational nodes responsive to the GPU memory consumption exceeding a threshold. The memory additionally includes managing the GPU memory consumption in accordance with the rewritten computational graph.