ML Runtime CPU Affinity and Niceness for Inference Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional ML model runtimes face challenges in maximizing throughput to meet service level agreements, suffer from inefficient CPU resource usage due to unnecessary context switches, and struggle with poor memory management and server crashes under peak loads.
Innovation Solution
A high-throughput ML model runtime that employs a master-worker architecture with Unix Domain Sockets and CPU affinity and niceness control to optimize process scheduling, using a multi-level token system for self-responsive rate limiting and efficient resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional ML model runtime processes are scheduled without CPU affinity control, then processor utilization occurs, but unnecessary context switches reduce throughput efficiency
Solution Approach 1:
The system segments processes into different priority levels using niceness values, creating distinct process groups (e.g., priority 0 for critical ML inference processes, priority 10 for background tasks). This segmentation allows the scheduler to efficiently manage context switches by prioritizing time-sensitive operations while still utilizing CPU resources for lower-priority tasks during idle periods.
Solution Approach 2:
The system dynamically adjusts process scheduling parameters (niceness values and CPU affinity masks) based on runtime conditions. When CPU capacity is available, lower niceness values are assigned to ML inference processes to maximize throughput. During peak loads, niceness values are adjusted to prevent system overload, optimizing the balance between throughput and system stability.
2Productivity
If ML model runtime increases processing capacity to meet throughput requirements, then service level agreement compliance improves, but system complexity and resource management difficulty increase
Solution Approach 1:
The system implements self-service through automated resource management mechanisms. The niceness adjustment system automatically monitors system load and dynamically modifies process priorities without manual intervention. CPU affinity masks are automatically configured based on process requirements, enabling the system to self-optimize throughput while maintaining manageable complexity through automation rather than manual configuration.
Solution Approach 2:
The system incorporates feedback loops that monitor throughput performance, system load, and resource utilization. Based on this feedback, the scheduling parameters are dynamically adjusted to maintain optimal performance. This closed-loop control enables the system to adapt to changing conditions and meet throughput requirements while preventing overload, thereby managing system complexity through intelligent feedback-driven adjustments.
3Adaptability or versatility
If traditional runtime systems process requests without rate limiting, then request handling flexibility is maintained, but memory management deteriorates and server crashes occur under peak loads
Solution Approach 1:
The system implements dynamic rate limiting that adapts to current system conditions rather than using fixed thresholds. The niceness values and CPU affinity configurations are adjusted in real-time based on memory utilization, CPU load, and request queue depth. This dynamic approach maintains request handling flexibility by allowing the system to accept variable loads while preventing memory exhaustion and server crashes through adaptive resource management.
Data Source
AI summary
Methods, systems, and computer-readable storage media for receiving, by a request processing engine of the ML model runtime, an inference request associated with a version model (VM), and determining, by the request processing engine that a VM-specific token and a global token are available for the inference request, the VM-specific token being available from a token pool that is specific to the VM, and in response: selecting a VM process (VMP) in a set of VMPs for execution of the inference request, the VMP being executed by a processor that is different from one or more processors executing one or more other VMPs in the set of VMPs, each VMP in the set of VMPs being specific to the VM and being designated for execution by a respective processor by a respective affinity setting, and providing the inference request to the VMP for execution.