Queue-Based GPU Virtualization for Resource Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current GPU resource management in computing systems results in low utilization due to dedicated allocation to individual users, leading to underutilization and high acquisition and operational costs, as users often do not utilize GPU devices for 100% of the allocated time, preventing sharing even when other users need access.
Innovation Solution
Implementing a queue-based GPU virtualization and management system that allows multiple users to share GPU resources by enqueuing tasks and scheduling their execution on a shared GPU device, either temporally or spatially, to optimize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If GPU devices are allocated to individual users on a dedicated basis, then each user has guaranteed access to GPU resources, but GPU utilization becomes extremely low and acquisition costs increase
Solution Approach 1:
The patent merges multiple dedicated GPU allocations into a shared GPU resource pool. Instead of assigning separate GPU devices to each user, the system combines GPU resources into a pool that serves multiple users through virtualization and queue-based task management, thereby improving utilization while maintaining access guarantees.
Solution Approach 2:
The patent makes a single GPU device universal by allowing it to serve multiple users and workloads simultaneously through temporal and spatial sharing. The GPU device performs multiple functions for different users at different times or concurrently on different cores, eliminating the need for dedicated per-user allocations.
2Productivity
If multiple users share GPU resources, then GPU utilization increases and costs decrease, but system complexity increases due to scheduling and virtualization requirements
Solution Approach 1:
The patent introduces an intermediary GPU resource manager that sits between users and physical GPU devices. This intermediary handles task queuing, scheduling, and resource allocation, simplifying the complexity by centralizing management logic rather than requiring complex peer-to-peer coordination among multiple users and devices.
Solution Approach 2:
The system implements self-service mechanisms where the GPU resource manager automatically queues, schedules, and allocates GPU resources without requiring manual intervention. Tasks are automatically enqueued when users submit workloads and dispatched when resources become available, reducing operational complexity.
3Quantity of substance
If GPU devices are statically allocated to users, then acquisition costs are high due to needing multiple devices, but dynamic sharing is prevented
Solution Approach 1:
The patent transitions from static GPU allocation to dynamic resource sharing. Instead of fixed assignments, the system dynamically assigns GPU resources to users based on current workload demands, task priority, and resource availability. This allows the same physical GPU devices to be flexibly allocated to different users at different times, reducing the total number of devices needed.
Data Source
AI summary
A method implemented by a server enables sharing of GPU resources by multiple clients. The server receives a request from a first client for GPU services. The request includes a first block of GPU code of an application executing on the first client. A first task corresponding to the first block of GPU code is enqueued in a task queue. The task queue includes a second task that corresponds to a second block of GPU code of an application executing on a second client. The server schedules a time for executing the first task using a GPU device that is assigned to the first client, and dispatches the first task to a GPU worker process to execute the first task at the scheduled time using the GPU device. The GPU device is shared, either temporally or spatially, by the first and second clients for executing the first and second tasks.