Runtime Memory Manager for GPU Multitenancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for sharing many-core devices like GPUs in heterogeneous clusters face limitations, including memory conflicts and the need for explicit device procurement, lacking transparency and dynamic binding/unbinding capabilities, especially when dealing with concurrent applications.
Innovation Solution
A system comprising a connection manager, virtual-many-cores, a dispatcher, and a memory manager that enables transparent mapping, controlled time-sharing, and dynamic binding of applications to many-core devices, with memory operations delayed until needed and a shadow memory on the host, allowing for efficient scheduling and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If CUDA runtime pre-allocates GPU memory to each CUDA context, then memory management is simplified, but memory utilization efficiency deteriorates due to inability to swap memory between CPU and GPU
Solution Approach 1:
The patent introduces a runtime system as an intermediary layer between applications and GPU devices. This runtime system manages memory allocation dynamically, allowing memory to be swapped between CPU and GPU as needed, thus improving memory utilization while maintaining ease of operation through automated management
Solution Approach 2:
The patent implements dynamic memory allocation where memory is not pre-allocated but allocated on-demand based on actual usage needs. The system can dynamically swap memory between CPU and GPU, transforming the static memory allocation model into a dynamic one that adapts to changing requirements
2Device complexity
If CUDA runtime serves CUDA calls from different applications in the order they arrive, then implementation is simple, but resource utilization deteriorates due to inability to prioritize or optimize execution order
Solution Approach 1:
The runtime system performs preliminary actions by pre-processing and queueing CUDA calls before execution. It can prioritize certain calls, batch similar operations, and optimize the execution order in advance, improving resource utilization without significantly increasing implementation complexity
Solution Approach 2:
The patent implements feedback mechanisms where the runtime system monitors GPU utilization and application requirements, then adjusts the scheduling and execution order of CUDA calls accordingly. This feedback loop enables optimized resource utilization while maintaining manageable system complexity
3Ease of operation
If explicit procurement of GPU devices is required by applications, then device allocation is straightforward, but system transparency deteriorates and application portability worsens
Solution Approach 1:
The runtime system acts as an intermediary that abstracts GPU device procurement from applications. Applications interact with the runtime system which handles device selection, allocation, and binding transparently, improving application portability while maintaining straightforward device allocation through the runtime's management interface
Solution Approach 2:
The patent segments the GPU procurement process into distinct layers: application layer (which requests resources), runtime layer (which manages allocation), and hardware layer (which provides resources). This segmentation allows applications to remain portable while the runtime handles device-specific allocation details
4Reliability
If a limited number of concurrent applications are supported on a GPU, then memory conflicts are avoided, but system scalability deteriorates
Solution Approach 1:
The patent introduces a new dimension of memory space by implementing a unified virtual memory system that spans both CPU and GPU memory. This allows concurrent applications to access memory from both domains, effectively increasing the available memory space and enabling more applications to run concurrently without memory conflicts
Solution Approach 2:
The runtime system implements a universal memory management mechanism that handles both CPU memory and GPU memory through a unified interface. This multi-functional memory system can allocate, swap, and manage memory across different domains, preventing memory conflicts while supporting scalable concurrency
Data Source
AI summary
The invention solves the problem of sharing many-core devices (e.g. GPUs) among concurrent applications running on heterogeneous clusters. In particular, the invention provides transparent mapping of applications to many-core devices (that is, the user does not need to be aware of the many-core devices present in the cluster and of their utilization), time-sharing of many-core devices among applications also in the presence of conflicting memory requirements, and dynamic binding/binding of applications to/from many-core devices (that is, applications do not need to be statically mapped to the same many-core device for their whole life-time).