GPU Virtualization Layer for Deep Learning Performance Isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current GPU sharing systems in deep learning clusters face challenges such as high integration and maintenance costs, inadequate performance isolation, and limited application compatibility, leading to inefficiencies and performance degradation in production settings.
Innovation Solution
A non-intrusive GPU sharing mechanism with a transparent virtualization layer that employs fine-grained thread-block level GPU kernel scheduling using slicing and preemption primitives, along with a priority-aware scheduler to ensure robust performance isolation and compatibility across diverse workloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPU sharing is implemented to improve resource utilization, then hardware utilization is enhanced, but performance isolation between high-priority and low-priority tasks deteriorates
Solution Approach 1:
The patent segments GPU execution into discrete kernel-level units and introduces fine-grained scheduling at the thread-block level within kernels. This segmentation enables the system to allocate specific thread-blocks to high-priority tasks while allowing low-priority tasks to utilize other thread-blocks or wait during critical periods, thereby maintaining performance isolation while improving overall hardware utilization.
Solution Approach 2:
The patent implements dynamic scheduling where the GPU can switch between serving high-priority and low-priority tasks based on real-time conditions. The system dynamically adjusts kernel execution timing, thread-block allocation, and scheduling decisions to ensure high-priority tasks meet their service level agreements while maximizing GPU utilization through opportunistic execution of low-priority tasks during idle periods.
2Reliability
If intrusive GPU sharing mechanisms are used to achieve performance isolation, then performance isolation is improved, but device complexity and integration cost increase
Solution Approach 1:
The patent introduces a transparent virtualization layer as an intermediary between applications and the physical GPU. This layer implements the complex scheduling and performance isolation logic without requiring modifications to user applications or deep integration into GPU hardware. The virtualization layer manages kernel transformations, thread-block scheduling, and priority enforcement, thereby achieving performance isolation while keeping the system architecture relatively simple and easy to integrate.
3Reliability
If fine-grained scheduling is implemented to improve performance isolation, then performance isolation is improved, but device complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where the scheduling system automatically transforms kernels and manages thread-block allocation without requiring manual configuration or complex external control. The transparent virtualization layer autonomously performs kernel analysis, transformation, and scheduling decisions based on task priorities, reducing the operational complexity despite the fine-grained nature of the scheduling.
Solution Approach 2:
The patent changes the scheduling parameter granularity from coarse (kernel-level) to fine (thread-block-level) while managing complexity through systematic approaches. By parameterizing the scheduling granularity and using automated transformation rules, the system achieves fine-grained control without proportionally increasing operational complexity, as the fine-grained scheduling is managed through standardized virtualization layer mechanisms.
Data Source
AI summary
A system and method are provided for facilitating graphics processing unit (GPU) sharing. The method includes providing a transparent virtualization layer positioned between one or more applications and one or more GPUs to be used by the one or more applications; intercepting device application programming interface (API) calls at the virtualization layer; and executing a task-agnostic scheduling algorithm to prioritize execution of tasks associated with the API calls, by the one or more GPUs, by prioritizing high-priority tasks while opportunistically scheduling kernels on the one or more GPUs from best effort tasks during GPU idle cycles.