Neural Network Execution Mode Selection Based on Memory Usage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large neural networks often exceed the memory capacity of a single Graphics Processing Unit (GPU), making it difficult to determine the optimal execution mode for efficient processing, as existing methods lack a dynamic mechanism to switch between modes based on memory usage.
Innovation Solution
A method that starts execution in Unified Memory mode, measures memory usage for each layer of the neural network, and dynamically selects the appropriate execution mode (normal, Out-of-Core, or Unified Memory) to avoid memory shortages and optimize performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If Unified Memory mode is used to support larger neural network models, then the model size capacity is improved, but the execution speed deteriorates
Solution Approach 1:
The system dynamically switches between execution modes (normal mode, OoC mode, UM mode) based on real-time memory usage measurements. This allows the system to adapt its execution characteristics - using UM mode when memory capacity is needed and switching to faster modes when memory constraints are satisfied, thereby resolving the speed-capacity tradeoff
Solution Approach 2:
The system changes execution parameters by selecting different execution modes based on measured memory usage. The mode selection is driven by parameter changes in memory consumption patterns, allowing the system to optimize between speed and capacity by adjusting the execution mode parameter
2Speed
If normal memory mode is used for execution, then the execution speed is improved, but the supported model size is limited
Solution Approach 1:
The system starts in UM mode for large models and dynamically switches to normal mode when memory usage measurements indicate that the model fits within GPU memory. This dynamic adaptation allows the system to support large models initially while automatically optimizing for speed when memory constraints are satisfied
3Quantity of substance
If Out-of-Core mode is used to support larger models than GPU memory, then the model size capacity is improved, but the execution speed deteriorates compared to UM mode
Solution Approach 1:
The system dynamically selects between OoC mode and UM mode based on measured memory usage. When measurements show that the model fits within GPU memory, the system switches to faster UM mode. When memory exceeds GPU capacity, it transitions to OoC mode, thereby adaptively resolving the speed-capacity contradiction
4Device complexity
If a fixed execution mode is selected before first execution, then the system complexity is reduced, but the execution efficiency deteriorates due to inability to adapt to actual memory usage
Solution Approach 1:
The system implements feedback by measuring actual memory usage during execution and using this information to select the optimal execution mode. This feedback mechanism allows the system to adapt to actual memory consumption patterns rather than relying on static predictions, thereby improving execution efficiency without excessive complexity
Solution Approach 2:
The system performs self-service by automatically measuring its own memory usage and selecting the appropriate execution mode without external intervention. This self-adjusting capability improves execution efficiency while keeping the control mechanism relatively simple
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer processing system are provided for selecting from among multiple Graphics Processing Unit (GPU) execution modes for a Neural Network (NN) having a size greater than a threshold size. The multiple GPU execution modes include a normal memory mode, an Out-of-Core (OoC) execution mode, and a Unified Memory (UM) mode. The method includes starting an execution on the NN with the UM mode and measuring the memory usage for each of layers of the NN. The method further includes selecting an execution mode based on the memory usage of all of the layers.