Worker Process Pinning for ML Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Real-time machine-learning applications experience inefficient throughput due to contention for processing unit time, leading to significant overhead and reduced efficiency in client-server architectures, particularly when multiple worker processes are migrated across different processing units.
Innovation Solution
Designate a subset of processing units as eligible for each worker process, ensuring each worker process executes exclusively on a single assigned unit, thereby reducing migration overhead and contention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple worker processes are deployed to increase throughput, then the number of requests that can be processed per second should increase linearly, but in practice the throughput degrades above 6 rps and latency exceeds 1 second after 17 rps
Solution Approach 1:
The system segments the set of processing units into dedicated subsets, with each worker process assigned to a specific processing unit. This segmentation prevents worker processes from migrating between processing units, eliminating the overhead associated with state transfer and cache invalidation, thereby maintaining high throughput without latency degradation.
2Ease of operation
If worker processes are migrated between processing units to balance load, then resource utilization may improve, but significant overhead is incurred due to transfer of application state and relevant data
Solution Approach 1:
The system performs preliminary assignment of worker processes to specific processing units before execution begins. By pre-establishing this mapping, the system eliminates the need for dynamic migration and associated state transfer overhead during runtime, thereby reducing energy loss while maintaining effective resource utilization.
3Ease of manufacture
If standard server implementations are used to deploy ML applications, then deployment is straightforward, but throughput is limited and significant overprovisioning of servers is required
Solution Approach 1:
The system applies a specialized scheduling approach specifically to worker processes of ML applications, designating certain processing units as eligible for these processes while allowing other processes to utilize all processing units. This local quality differentiation optimizes throughput for ML workloads without complicating the overall system deployment.
Data Source
AI summary
A server having multiple processing units executes a machine-learning application that instantiates a plurality of worker processes that accept connections from client devices at a socket. To prevent excess contention and migration of the worker processes to different processing units, each worker process is specified to a subset of the processing units eligible to execute that respective worker process. The respective subsets for each worker process may be mutually exclusive and each worker process may be assigned a single processing unit eligible to execute that worker process. This enables significantly higher throughput for the worker processes by preventing normal process migration across the plurality of processing units.