GPU Video Memory Allocation Using Host Memory Overflow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning frameworks like TensorFlow and PyTorch limit computational efficiency due to the physical video memory capacity of GPUs, restricting the deployment and use of machine learning models.
Innovation Solution
A method and apparatus that dynamically allocate video memory by generating a second call request for a memory hosting interface when the physical video memory capacity is exceeded, allowing for virtual memory allocation through interfaces like cudaMallocManaged.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the physical video memory capacity of GPU cards is used to allocate memory for machine learning tasks, then the memory allocation is simple and direct, but the computational efficiency and use effect of machine learning are significantly impacted due to limited capacity
Solution Approach 1:
The patent introduces a memory hosting interface as an intermediary between the application program and the physical video memory. When the video memory allocation interface cannot meet the memory demand, the system automatically generates a call to the memory hosting interface to obtain additional memory from the host system, thereby resolving the contradiction between limited video memory capacity and the need for higher computational efficiency.
2Ease of operation
If the video memory allocation interface is used to apply for video memory, then the allocation process is straightforward, but it results in significant impact on computational efficiency and use effect of machine learning due to physical video memory capacity limitations
Solution Approach 1:
The memory hosting interface serves as a mediator that maintains the simplicity of the allocation process while overcoming physical memory limitations. The system transparently handles the complexity of memory management by automatically switching between video memory and host memory through the hosting interface, preserving ease of operation for developers while significantly improving computational efficiency.
Solution Approach 2:
The memory hosting interface provides multi-functionality by serving both as a supplement to video memory and as an alternative memory source. It enables the system to handle both small-scale tasks using only video memory and large-scale tasks by combining video memory with host memory, thereby improving computational efficiency across different workloads while maintaining a unified allocation interface.
3Quantity of substance
If the physical video memory capacity is exceeded, then the system cannot allocate additional memory through the video memory allocation interface, but this limits the deployment and use scope of machine learning models
Solution Approach 1:
The memory hosting interface acts as a bridge that extends available memory beyond physical video memory capacity. When video memory is exhausted, the system automatically invokes the hosting interface to allocate additional memory from the host system, thereby eliminating the hard limit imposed by physical video memory and expanding the deployment scope of machine learning models to include larger models and datasets.
Solution Approach 2:
The system dynamically adjusts memory allocation by switching between video memory and host memory based on available capacity and task requirements. This dynamic memory management enables the system to adapt to varying memory demands, allowing deployment of diverse machine learning models with different memory requirements without being constrained by fixed video memory capacity.
Data Source
AI summary
A video memory allocation method and apparatus, and a medium and an electronic device are provided. The method includes: determining a usage capacity of a physical video memory in response to receiving a first call request of an application program for a video memory allocation interface in a target video memory management link library; if a sum of the usage capacity and an applied capacity corresponding to the first call request is greater than a total capacity of the physical video memory, generating a second call request for calling a memory hosting interface according to the first call request, and calling the memory hosting interface based on the second call request; and obtaining a response of the first call request according to a calling result of the second call request.