Multi-Tenant Job Scheduling for Fair Throughput Under Pipeline Pressure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud service providers face challenges in ensuring multi-tenant fairness by preventing resource contention and bottleneck formation due to spikes in job requests or large data sets, as existing resource-sharing models fail to account for dynamic resource availability and can be dominated by a small number of tenants.
Innovation Solution
Implementing a multi-tenant fairness job scheduler that calculates historical and target throughputs based on base throughput, system pressure, and tenant weight, using a stream and database-based system to determine whether to process or throttle job requests, ensuring equitable resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a small number of tenants dominate the job request pipeline, then those tenants can process more jobs, but resource contention increases and bottlenecks occur for other tenants
Solution Approach 1:
The system continuously monitors the number of in-flight jobs for each tenant and compares it against a calculated fair share threshold. This feedback mechanism dynamically adjusts job acceptance decisions based on current system state, preventing any single tenant from dominating the pipeline while maintaining high overall throughput.
Solution Approach 2:
The fair share threshold is not static but dynamically calculated based on real-time factors including base throughput, system pressure, tenant weight, and number of active tenants. This dynamic adjustment allows the system to adapt to changing conditions and maintain fair resource allocation across different workload scenarios.
2Adaptability or versatility
If the system processes all incoming job requests, then tenant demand is met, but system pressure increases causing bottlenecks
Solution Approach 1:
The system deliberately processes only a fair share of incoming job requests rather than all requests. By accepting partial action (rejecting some jobs when thresholds are exceeded), the system prevents overwhelming itself and maintains stable operation under varying demand conditions.
Solution Approach 2:
The system proactively rejects job requests before system pressure becomes critical. By comparing incoming jobs against the fair share threshold in advance, the system prevents bottleneck formation rather than reacting after congestion occurs.
3Reliability
If the system rejects job requests to maintain fairness, then resource contention is reduced, but tenant service level may deteriorate
Solution Approach 1:
The system uses tenant weight as a configurable parameter to adjust the fair share threshold for different tenants. This allows tenants with higher priority or paid plans to receive a larger portion of processing resources, maintaining fairness while accommodating different service level requirements.
Solution Approach 2:
The system applies different fair share thresholds to different tenants based on their individual weights and characteristics. Rather than uniform treatment, each tenant receives a customized allocation that reflects their service agreement and system priorities.
Data Source
AI summary
Techniques are described for determining whether to process a job request. An example, method can include a device receiving a first message from a first stream, the first message comprising a job request from a tenant and a tenant identifier. The device can detect a base number of units permissible to be processed for the tenant over a unit of time. The device can detect a processing speed of a downstream processor of an asynchronous pipeline. The device can detect a number of messages in a second stream, the downstream processor configured to receive messages from the second stream. The device can determine a target throughput and a historical throughput for the tenant. The device can compare the target throughput with the historical throughput to determine whether to process the job request. The device can schedule the job request for processing based at least in part on the comparison.