Multi-Layer Fair Share Scheduling for Rate Limiting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-tenant computing environments, a single tenant's intensive computing tasks can disproportionately consume shared resources, leading to performance delays and availability issues for other tenants, as existing methods fail to effectively balance resource usage and manage workload distribution.
Innovation Solution
A dynamic rate limiting mechanism is implemented using a token-based system and multi-layer queuing, where tokens are allocated based on estimated workloads, and requests are routed through multiple queues with varying maximum workloads and prioritization schemes to prevent resource overconsumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single tenant executes computationally expensive operations or a large number of operations within a short period, then the tenant's processing speed and productivity are improved, but the shared compute resources are consumed excessively, causing performance delays and availability issues for other tenants
Solution Approach 1:
The patent segments the compute resources and workload management into multiple independent queues (first queue, second queue, third queue) with different rate limiting parameters. Each queue has its own maximum workload threshold, allowing the system to segment tenant workloads into different processing channels based on intensity, thereby preventing any single tenant from monopolizing all shared resources while maintaining high productivity for legitimate workloads
Solution Approach 2:
The patent dynamically changes the parameter of maximum workload thresholds across different queues. The first queue has a lower threshold for intensive operations, the second queue has a medium threshold, and the third queue has a higher threshold. This parameter variation allows the system to adapt resource allocation based on workload characteristics, ensuring that no single tenant can exceed acceptable resource consumption limits while still allowing high productivity when resources are available
2Ease of operation
If rate limiting is implemented to prevent resource overconsumption, then resource allocation fairness is improved, but request processing time and queue wait times increase
Solution Approach 1:
The patent applies preliminary action by establishing multiple queues with pre-configured rate limiting thresholds before workloads arrive. The system pre-defines the first queue with a lower maximum workload threshold for intensive operations, the second queue with a medium threshold, and the third queue with a higher threshold. This preliminary structuring allows requests to be rapidly routed to appropriate queues without dynamic calculation delays, maintaining fairness while minimizing processing time
Solution Approach 2:
The patent implements dynamics by allowing the system to adaptively route workloads to different queues based on current resource availability and tenant requirements. The multi-queue structure enables dynamic adjustment of rate limiting application - more stringent limits on the first queue for intensive operations, moderate limits on the second queue, and more flexible limits on the third queue - thereby balancing fairness with efficient request processing
3Use of energy by moving object
If multi-tenant computing is used to share computing power, then resource utilization efficiency is improved, but a single tenant can disproportionately consume shared resources, leading to performance degradation for other tenants
Solution Approach 1:
The patent segments the shared compute resources into multiple isolated queues, each with its own rate limiting enforcement. The first queue handles intensive operations with strict limits, the second queue handles medium-intensity operations with moderate limits, and the third queue handles standard operations with more flexible limits. This segmentation prevents any single tenant from consuming disproportionate resources while maintaining high overall utilization of the shared compute infrastructure
Solution Approach 2:
The patent introduces the multi-queue system as an intermediary layer between tenant requests and the shared compute resources. This intermediary structure monitors and controls workload submission, applying appropriate rate limiting based on the queue the request is routed to. The intermediary prevents harmful resource overconsumption by individual tenants while allowing efficient resource utilization across the multi-tenant environment
Data Source
AI summary
The embodiments disclosed herein relate to predictive rate limiting. A workload for completing a request is predicted based on, for example, characteristics of a ruleset to be applied and characteristics of a target set upon which the ruleset is to be applied. The workload is mapped to a set of tokens or credits. If a requestor has sufficient tokens to cover the workload for the request, the request is processed. The request may be processed in accordance with a set of processing queues. Each processing queue is associated with a maximum per-tenant workload. A request may be added to a processing queue as long as adding the request does not result in exceeding the maximum per-tenant workload. Requests within a processing queue may be processed in a First In First Out (FIFO) order.