Dynamic GPU Sharing for AI Services
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies for executing multiple AI services on a GPU fail to fully utilize GPU resources, leading to inefficiencies in memory and compute core usage.
Innovation Solution
A method and apparatus for dynamically optimizing GPU sharing policies by detecting resource requirements for AI services, allowing for the optimal combination and execution of multiple AI services on a single GPU.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple AI services are executed on a single GPU using static sharing methods, then the GPU can be shared by multiple virtual infrastructures, but the GPU utilization rate remains low due to insufficient dynamic optimization
Solution Approach 1:
The patent implements dynamic GPU sharing policies that adjust resource allocation based on real-time workload characteristics. The system monitors AI service performance metrics and dynamically modifies sharing configurations, transitioning from static pre-configured sharing to adaptive dynamic sharing that optimizes GPU utilization while maintaining multi-service execution capability
Solution Approach 2:
The system changes key parameters including batch size, number of simultaneous AI services, memory allocation, and compute core distribution based on detected workload patterns. By dynamically adjusting these parameters, the system achieves optimal GPU utilization rates while supporting multiple AI services with different resource requirements
2Adaptability or versatility
If GPU context switching is implemented to support multiple AI services, then service diversity is improved, but the cost of context switching is high due to large GPU context size
Solution Approach 1:
The patent segments the GPU resources into multiple virtual GPUs, each dedicated to specific AI services or service groups. This segmentation allows independent management of GPU contexts for different services, eliminating the need for frequent full-context switching while maintaining multi-service execution capability. Each vGPU maintains its own context state, reducing context switch overhead
Solution Approach 2:
The system introduces virtual GPU instances as intermediaries between physical GPU resources and AI services. These vGPUs act as context containers that can be efficiently created, destroyed, and switched without the high cost of direct GPU context switching. The virtualization layer manages context persistence and restoration, reducing the time penalty associated with context transitions
3Ease of manufacture
If AI services are executed with fixed resource allocation, then deployment simplicity is maintained, but resource utilization efficiency decreases due to mismatched allocation
Solution Approach 1:
The system transitions from fixed static resource allocation to dynamic allocation that adapts to actual AI service workload characteristics. Resource allocation parameters such as memory size, compute core count, and batch processing size are automatically adjusted based on real-time performance monitoring and workload analysis, maintaining deployment simplicity while significantly improving utilization efficiency
Solution Approach 2:
The patent implements feedback mechanisms that continuously monitor AI service performance metrics including throughput, latency, and resource consumption. This feedback is used to automatically adjust resource allocation decisions, creating a closed-loop system that optimizes resource distribution without requiring complex manual configuration or deployment procedures
Data Source
AI summary
Disclosed herein is a method for executing Artificial Intelligence (AI) services based on virtual infrastructures. The method includes configuring the sharing type of a computational processing unit, executing an AI service based on a virtual infrastructure using requirements for the AI service and information about the sharing type of the computational processing unit, and performing optimization for the AI service.