Adaptive Throttling for Multi-Tenant Cloud Rate Limits

Solution Overview

Problem

Cloud Service Providers face challenges in managing finite computing resources effectively, leading to underutilization and poor performance due to fixed request quotas that restrict tenants from consuming more resources than allocated, which can result in stress on the cloud computing system.

Innovation Solution

Implementing an adaptive throttling service that manages concurrent rate limits using token buckets, allowing tenants to access additional processing capacity when the system is underutilized and enforcing limits during stress conditions, thereby dynamically adjusting resource allocation based on demand.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a predefined rate limit is specified on the number of concurrent requests from a tenant, then resource management is simplified and system stability is maintained, but resource utilization decreases and tenant performance suffers due to underutilization

Engineering Contradiction:

Improvesystem stabilityVSAvoidresource utilization

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent implements dynamic rate limiting by transitioning from static predefined limits to adaptive thresholds that automatically adjust based on real-time system conditions. The throttling service monitors system metrics (CPU utilization, memory availability, request queue depth) and dynamically modifies the number of allowed concurrent requests per tenant, enabling the system to maintain stability while maximizing resource utilization during varying load conditions

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes the parameter of rate limits from fixed values to variable values that are continuously adjusted based on system state. The throttling service modifies the concurrent request threshold parameter dynamically, increasing it when system resources are abundant and decreasing it when resources are constrained, thereby resolving the contradiction between stability and utilization

Inventive Principle:

Principle #35Parameter changes

2Reliability

If a predefined rate limit is specified on the number of concurrent requests from a tenant, then system overload is prevented, but tenant performance decreases when additional processing capacity is available

Engineering Contradiction:

Improvesystem overload preventionVSAvoidtenant performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent implements dynamic rate limiting by transitioning from static predefined limits to adaptive thresholds that automatically adjust based on real-time system conditions. The throttling service monitors system metrics (CPU utilization, memory availability, request queue depth) and dynamically modifies the number of allowed concurrent requests per tenant, enabling the system to maintain stability while maximizing resource utilization during varying load conditions

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system incorporates feedback loops where the throttling service continuously monitors system performance metrics and uses this information to adjust rate limits. When system resources are underutilized, the service increases allowed concurrent requests; when resources are strained, it decreases the limit. This feedback mechanism ensures tenant performance scales with available capacity while preventing overload

Inventive Principle:

Principle #23Feedback

3Ease of operation

If fixed request quotas are allocated to tenants, then fair distribution of resources is ensured, but resource utilization decreases and performance suffers due to restrictions on consuming available capacity

Engineering Contradiction:

Improvefair distributionVSAvoidresource utilization

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The patent implements dynamic rate limiting by transitioning from static predefined limits to adaptive thresholds that automatically adjust based on real-time system conditions. The throttling service monitors system metrics (CPU utilization, memory availability, request queue depth) and dynamically modifies the number of allowed concurrent requests per tenant, enabling the system to maintain stability while maximizing resource utilization during varying load conditions

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The throttling service implements a universal resource management mechanism that serves multiple functions: it ensures fair distribution when resources are constrained, allows tenants to access additional capacity when resources are abundant, and maintains system stability under all conditions. This multi-functional approach replaces the need for separate fixed quota systems

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20230281050A1Adaptive throttling with tenant-based concurrent rate limits for a multi-tenant system

Publication Date: 2023.09.07 ORACLE INT CORP

AI summary

The present embodiments relate to adaptive throttling with tenant-based concurrent rate limits. A first exemplary embodiment provides a method for adaptive throttling with tenant-based concurrent rate limits. The method can include a computing device receiving a request directed to a first tenant of a multi-tenant cloud infrastructure system. The first tenant being granted access to a limited processing capacity to process a limited number of requests. The computing device can further determine whether the multi-tenant cloud infrastructure system is in stress. The computing device can further permit the first tenant access to the additional processing capacity to process a number of requests greater than the limited number of requests.