GPU Memory Virtualization via Scheduler Module
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems lack effective management and virtualization of GPU memory, leading to resource interference and efficiency issues when multiple processes or API requests compete for GPU resources, resulting in potential out-of-memory errors and scheduling challenges.
Innovation Solution
Implementing a method and system for virtualizing GPU memory through a scheduler module that allocates and manages GPU resources by assigning API requests to specific slots on virtual machines, monitoring memory usage, and dynamically adjusting resource allocation to prevent overutilization and ensure fair access to GPU memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If GPU memory is not virtualized or managed, then each process can access all GPU memory resources, but this leads to resource interference and out-of-memory errors when multiple processes run simultaneously
Solution Approach 1:
The patent introduces a GPU virtualization layer as an intermediary between processes and physical GPU memory. This virtualization layer manages memory allocation, tracks usage by each process, and prevents any single process from exceeding its allocated portion, thereby eliminating out-of-memory errors while maintaining manageable complexity through automated tracking and allocation mechanisms.
Solution Approach 2:
The patent divides the total GPU memory into separate allocated portions for each process. The system tracks and enforces memory limits per process, segmenting the shared physical memory resource into isolated virtual memory spaces. This segmentation prevents resource interference between processes while maintaining overall system reliability.
2Productivity
If GPU memory is allocated to multiple processes simultaneously, then resource utilization improves, but it becomes difficult to enforce actual limits on memory usage per process
Solution Approach 1:
The patent implements a feedback mechanism where the GPU virtualization layer continuously monitors memory usage by each process against its allocated limits. When a process approaches or exceeds its allocated portion, the system detects this through tracking data structures and enforces limits by preventing further allocations or triggering evictions, making limit enforcement automatic and efficient without manual intervention.
Solution Approach 2:
The virtualization layer automatically manages memory allocation and limit enforcement without requiring manual configuration for each process. The system self-regulates by tracking usage patterns, allocating memory portions dynamically, and enforcing limits through automated mechanisms, thereby improving ease of operation while maintaining high resource utilization.
3Productivity
If GPU memory is not freed after request completion, then active processing is simplified, but this leads to resource accumulation and interference with other users over time
Solution Approach 1:
The patent implements periodic cleanup actions where the GPU virtualization layer regularly checks for completed requests and frees their allocated memory portions. The system uses timing mechanisms to identify when requests have completed processing and automatically releases their memory allocations back to the pool, preventing long-term accumulation while maintaining efficient active processing through deterministic memory management.
Solution Approach 2:
The patent implements a mechanism where memory allocated to completed requests is automatically discarded and recovered. The virtualization layer tracks the lifecycle of each allocation, and upon request completion, it discards the process's access rights and recovers the physical memory back to the available pool, preventing resource accumulation and ensuring sustained system productivity over time.
Data Source
AI summary
The technology includes methods, processes, and systems for virtualizing graphics processing unit (GPU) memory. Example embodiments of the technology include managing an amount of GPU memory used by one or more processes, such as Application Programming Interfaces (APIs), that directly or indirectly impact one or more other processes running on the same GPU. Managing and/or virtualizing the amount of GPU memory may ensure that an end user does not receive a GPU out-of-memory error because the API request is impacted by the processing of other API requests. A virtual machine with access to a GPU may be organized with one or more job slots that are configured to specify the number of processes that are able to run concurrently on a specific virtual machine. A process may be configured on each virtual machine running a software program or API and is used to schedule work based on GPU memory requirements.