Throttling Queue for Multi-Tenant Request Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional request scheduling and processing systems in distributed computing environments face challenges in ensuring fair resource allocation across multiple tenants, particularly when handling surges in requests, leading to resource overload and unfair consumption, which can result in dropped requests and poor system availability.
Innovation Solution
Implementing a throttling queue system that redirects requests exceeding defined thresholds to a prioritized queue, while using dominant resource scheduling and hard limits on resource utilization to manage resource allocation and prevent overconsumption by any single tenant account, thereby ensuring fair and efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If requests are processed from account queues on a FIFO basis, then processing order is simple and predictable, but resource overload occurs when request surges exceed system capacity
Solution Approach 1:
The system segments the request queue into two distinct queues: account queues for normal FIFO processing and a throttling queue for overflow requests. This segmentation allows the system to maintain simple FIFO processing for normal operations while isolating surge requests to prevent resource overload and maintain system availability.
Solution Approach 2:
The throttling queue acts as an intermediary buffer between incoming request surges and the account queues. When account queues approach capacity, requests are redirected to the throttling queue, which processes them separately with higher priority, preventing resource overload while maintaining overall system availability.
2Productivity
If all incoming requests are accepted and queued, then no requests are dropped, but system resources are consumed and fairness among tenants deteriorates
Solution Approach 1:
The system extracts requests that would cause resource overload from the normal account queues and places them in a separate throttling queue. This extraction mechanism prevents unfair resource consumption by isolating surge requests while maintaining fair resource allocation for normal requests across multiple tenants.
Solution Approach 2:
The system changes the processing parameters for different queue types: account queues use standard FIFO processing with fair resource allocation, while the throttling queue uses priority processing with exclusive resource access. This parameter differentiation allows high throughput for normal requests while ensuring fairness among tenants.
3Reliability
If requests exceeding thresholds are dropped to a throttling queue, then resource overload is prevented, but processing complexity increases
Solution Approach 1:
The system segments request management into two distinct queue structures with different processing rules. Account queues handle normal requests with simple FIFO ordering, while the throttling queue handles overflow requests with priority processing. This segmentation manages resource utilization effectively while keeping each queue's management logic relatively simple.
Solution Approach 2:
The system dynamically routes requests to appropriate queues based on current resource utilization thresholds. When account queues approach capacity, the routing logic dynamically redirects requests to the throttling queue. This dynamic adaptation prevents resource overload while maintaining manageable queue structures through context-aware routing decisions.
Data Source
AI summary
Various methods and systems for implementing request scheduling and processing in a multi-tenant distributed computing environment are provided. Requests to utilize system resources in the distributed computing environment are stored in account queues corresponding to tenant accounts. If storing a request in an account queue would exceed a throttling threshold such as a limit on the number of requests stored per account, the request is dropped to a throttling queue. A scheduler prioritizes processing requests stored in the processing queue before processing requests stored in the account queues. The account queues can be drained using dominant resource scheduling. In some embodiments, a request is not picked up from an account queue if processing the request would exceed a predefined hard limit on system resource utilization for the corresponding tenant account. In some embodiments, the hard limit is defined as a percentage of threads the system has to process requests.