Unified Memory Allocation in Multitenant GPU Environments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multitenant GPU environments, existing memory management techniques lead to Out-of-Memory Errors and inefficient resource utilization due to premature memory deallocation and inadequate scheduling of tasks based on available memory.
Innovation Solution
A method that delays memory deallocation by identifying and deallocating only unnecessary memory usage within tasks, and utilizes unified memory to allow tasks to exceed the GPU's total memory amount, thereby enhancing memory allocation and scheduling efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If memory is deallocated just before program termination, then memory management is simple, but GPU resource utilization is low and other tasks cannot be scheduled
Solution Approach 1:
The patent segments memory management from program-level to variable-level by tracking each variable's usage status individually. The compiler analyzes def-use chains to determine when each variable is no longer needed, allowing selective deallocation of individual variables rather than waiting for program termination. This enables finer-grained memory control and improves GPU resource utilization.
Solution Approach 2:
The system implements feedback mechanisms where the compiler continuously monitors variable usage status during task execution. When a variable's usage is determined to be finished through def-use chain analysis, the system automatically triggers deallocation and updates the scheduler about available memory, enabling dynamic memory management that responds to actual usage patterns.
2Ease of operation
If memory is held until program termination, then memory allocation is simple, but Out-of-Memory Errors occur in multitenant environments
Solution Approach 1:
The system performs preliminary analysis of variable usage patterns using def-use chain analysis during compilation. This allows the system to pre-determine when each variable will no longer be needed and schedule its deallocation accordingly, preventing memory exhaustion before it occurs. The scheduler uses this information to proactively manage memory across multiple tenants.
Solution Approach 2:
The compiler provides feedback about variable usage status to the memory management system, enabling automatic deallocation decisions. This feedback mechanism tracks whether each variable is still needed by analyzing its definition and usage chains, ensuring memory is released only when safe to do so, thereby preventing Out-of-Memory Errors in multitenant GPU environments.
3Ease of operation
If tasks are scheduled based on current extra memory only, then scheduling is straightforward, but tasks with slightly higher memory requirements suffer long pending times
Solution Approach 1:
The patent transforms static memory allocation into a dynamic system where memory availability changes continuously during task execution. As variables are deallocated based on their usage status, the available memory pool dynamically increases, allowing the scheduler to flexibly allocate memory to pending tasks without requiring them to wait for program termination or fixed time intervals.
Solution Approach 2:
The system implements continuous feedback between the compiler, memory management, and scheduler. The compiler monitors variable usage and provides real-time information about deallocatable memory to the scheduler, which then adjusts task scheduling decisions accordingly. This feedback loop eliminates long pending times by enabling immediate task scheduling when memory becomes available.
4Quantity of substance
If unified memory is used to exceed GPU total memory amount, then memory capacity is increased, but memory management complexity increases
Solution Approach 1:
The patent introduces the compiler as an intermediary that automatically performs def-use chain analysis and determines variable deallocation timing. This intermediary handles the complexity of tracking variable usage across multiple tasks and tenants, freeing the memory management system from complex manual tracking while enabling unified memory to be used beyond the GPU's physical memory limits.
Solution Approach 2:
The system implements self-service memory management where the compiler automatically analyzes and determines when variables can be deallocated without external intervention. By embedding the analysis logic in the compilation process, the system autonomously manages unified memory allocation and deallocation, reducing the perceived complexity for users while enabling extended memory capacity.
Data Source
AI summary
Provided is a memory allocation/deallocation method for improving utilization and throughput of GPU, and a method for allocating a memory using a unified memory according to an embodiment may include: checking whether a kernel of an executed task is terminated; checking whether there is a variable in which the use of the memory is finished after completing kernel execution among variables used as a kernel factor of the executed task; deallocating, when there is the variable in which the use of the memory is finished, the corresponding variable; calculating the deallocated memory amount; and transmitting the deallocated memory amount to a scheduler.