Inference Workload Affinity Scheduling for GPU Latency Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to efficiently schedule concurrent execution of multiple artificial intelligence inference workloads on graphics processing units (GPUs) without causing latency issues, leading to suboptimal resource utilization and performance.
Innovation Solution
A system identifies inference workload instances with affinity for concurrent execution on a GPU's core processing unit by measuring resource requirements and latency, allowing models with compatible resource demands to run simultaneously, while preventing models that would exceed latency limits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple inference workload instances are scheduled to run concurrently on the same GPU core processing unit, then resource utilization and productivity improve, but latency may exceed acceptable limits causing performance degradation
Solution Approach 1:
The system dynamically changes scheduling parameters by computing affinity scores based on latency measurements between different inference workload pairs. Workloads are scheduled concurrently only when their pairwise latency remains below acceptable thresholds, allowing the system to adapt concurrency levels based on real-time performance characteristics
Solution Approach 2:
The system performs preliminary latency measurements and affinity computations before scheduling inference workloads concurrently. By pre-evaluating the compatibility of workload pairs and storing affinity information, the system avoids scheduling conflicts that would cause latency violations, enabling safe concurrent execution
2Ease of operation
If inference workloads are scheduled without affinity awareness, then scheduling simplicity is maintained, but resource utilization efficiency decreases
Solution Approach 1:
The system implements self-service scheduling by automatically computing affinity scores and making scheduling decisions based on measured latency characteristics. The affinity module autonomously evaluates workload pairs and determines concurrent schedulability without requiring manual intervention or complex external orchestration
Solution Approach 2:
The system uses feedback from latency measurements to improve scheduling decisions. By continuously monitoring the actual latency impact of concurrent workload pairs and updating affinity information, the system learns from past scheduling outcomes to optimize future resource allocation and maximize utilization efficiency
Data Source
AI summary
A computer program product provides program instructions executable by a processor to cause the processor to perform operations. The operations include identifying a first inference workload instance using a first inference model and a second inference workload instance using a second inference model and identifying whether the first and second inference models have affinity for being run concurrently on a core processing unit. The first and second inference models have affinity if the first and second inference workload instances can run concurrently on the core processing unit without causing either of the first and second inference workload instances to experience latency above a predetermined limit. The operations further include causing the first and second inference workload instances to be run concurrently on the core processing unit if the first and second inference models have been identified to have affinity for being run concurrently on the core processing unit.