Work-Based Token Throttling for Service Request Admission Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems that provide software-as-a-service or Web-based services face challenges in managing service requests due to varying workloads, as existing overload management techniques, such as threshold-based request throttling, fail to account for differences in the amount of work required for different types of requests, leading to inefficient resource allocation and potential service degradation.
Innovation Solution
A computing system dynamically adjusts its maximum request rate based on current and committed work throughput rates, using token-based admission control mechanisms like the 'leaky bucket' approach to manage service requests, ensuring that the system operates within acceptable performance levels by throttling requests when overloaded and raising rates when under-loaded, while supporting multiple admission control mechanisms to adapt to changing workloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a fixed or varying global threshold is used to throttle service requests, then the system can prevent overload by denying service when the threshold is crossed, but the system cannot account for differences in the amount of work required for different types of requests, leading to inefficient resource allocation
Solution Approach 1:
The patent changes the parameter from a fixed request rate threshold to a dynamic work-based threshold. Instead of throttling based solely on the number of requests per second, the system adjusts the threshold based on the actual work required to process different request types. This allows the system to maintain reliable overload prevention while improving resource allocation efficiency by considering the varying computational cost of different operations.
Solution Approach 2:
The patent introduces dynamics by making the threshold variable rather than fixed. The threshold dynamically adapts to the current workload characteristics and request types. This dynamic approach enables the system to respond to changing conditions, allowing more requests when work is light and throttling more aggressively when work is intensive, thus balancing reliability with productivity.
2Device complexity
If a single global threshold is used for request throttling, then the system can implement simple overload protection, but it is difficult to define a meaningful threshold in a system that receives different types of requests at varying, unpredictable rates
Solution Approach 1:
The patent applies segmentation by dividing the monolithic global threshold into multiple per-request-type thresholds. Instead of using a single threshold for all requests, the system creates separate thresholds for different request types (e.g., read requests, write requests, query requests). This segmentation maintains relative simplicity while enabling the system to handle varying request types effectively, as each threshold can be independently tuned and adjusted.
Solution Approach 2:
The patent implements local quality by assigning different threshold characteristics to different request types. Each request type receives a customized threshold based on its specific work requirements and impact on the system. This allows the throttling mechanism to adapt to the unique properties of each request type, improving versatility while maintaining manageable complexity through localized parameter adjustment.
3Reliability
If the system throttles requests aggressively to prevent overload, then system availability is maintained, but legitimate requests may be denied due to varying workload conditions
Solution Approach 1:
The patent incorporates feedback mechanisms that continuously monitor system workload and request characteristics. This feedback allows the system to adjust thresholds dynamically based on actual conditions. When the system is lightly loaded, thresholds are raised to allow more requests, improving ease of operation. When overload is detected, thresholds are lowered to protect availability. This feedback loop resolves the contradiction by adapting behavior to current system state.
Solution Approach 2:
The patent uses dynamic threshold adjustment to balance availability and operational smoothness. Rather than using static, overly conservative thresholds that constantly deny requests, the system dynamically adapts thresholds to current workload conditions. This dynamic behavior allows the system to maintain availability when necessary while providing smooth, uninterrupted service during normal operating conditions, thus resolving the contradiction between reliability and ease of operation.
Data Source
AI summary
A system that provides services to clients may receive and service requests, various ones of which may require different amounts of work. An admission control mechanism may manage requests based on tokens, each of which represents a fixed amount of work. The tokens may be added to a token bucket at rate that is dependent on a target work throughput rate while the number of tokens in the bucket does not exceed its maximum capacity. If at least a pre-determined minimum number of tokens is present in the bucket when a service request is received, it may be serviced. Servicing a request may include deducting an initial number of tokens from the bucket, determining that the amount of work performed in servicing the request is different than that represented by the initially deducted tokens, and deducting additional tokens from or replacing tokens in the bucket to reflect the difference.