Dynamic Request Throttling for Non-Uniform Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems that provide software-as-a-service face challenges in managing service requests due to varying workloads, as existing overload prevention methods, such as threshold-based request throttling, fail to account for differences in request types and rates, leading to inefficient resource allocation and performance issues.
Innovation Solution
A computing system dynamically adjusts the maximum request rate based on current and committed work throughput rates, using token-based admission control mechanisms like the 'leaky bucket' approach to manage service requests, ensuring efficient resource allocation and maintaining acceptable performance levels even under non-uniform workloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a fixed or varying global threshold is used to throttle service requests, then the system can prevent overload by blocking requests, but the system cannot distinguish between different types of requests with varying workloads, leading to inefficient resource allocation
Solution Approach 1:
The patent applies local quality by transitioning from a uniform global threshold to per-client or per-request type thresholds. Each client is assigned a customized threshold based on their historical behavior, workload patterns, and service requirements. This allows the system to differentiate between clients with different workload characteristics, enabling more precise request throttling that adapts to local variations in request patterns while maintaining overall system reliability.
2Device complexity
If a single global threshold is used for request throttling, then the system structure remains simple, but the system cannot adapt to varying and unpredictable request rates and work requirements
Solution Approach 1:
The patent implements dynamics by making the threshold values adaptive rather than static. The system continuously monitors client behavior, request patterns, and system performance metrics to dynamically adjust individual client thresholds in real-time. This dynamic adjustment mechanism allows the throttling system to respond to changing workload conditions, unpredictable request rates, and evolving service requirements without requiring complex manual reconfiguration.
3Device complexity
If requests are throttled based on request rate alone, then the throttling mechanism is simple to implement, but the system cannot account for differences in the amount of work required to satisfy different request types
Solution Approach 1:
The patent applies parameter changes by expanding the throttling decision criteria from单一的 request rate parameter to multiple parameters including request type, historical workload patterns, average processing time, and system current state. The system calculates a composite threshold that incorporates these varying parameters, allowing it to account for the different amounts of work required by various request types. This multi-parameter approach improves resource utilization efficiency by throttling based on actual work burden rather than just request frequency.
Data Source
AI summary
A system that provides services to clients may receive and service requests, various ones of which may require different amounts of work. The system may determine whether it is operating in an overloaded or underloaded state based on a current work throughput rate, a target work throughput rate, a maximum request rate, or an actual request rate, and may dynamically adjust the maximum request rate in response. For example, if the maximum request rate is being exceeded, the maximum request rate may be raised or lowered, dependent on the current work throughput rate. If the target or committed work throughput rate is being exceeded, but the maximum request rate is not being exceeded, a lower maximum request rate may be proposed. Adjustments to the maximum request rate may be made using multiple incremental adjustments. Service request tokens may be added to a leaky token bucket at the maximum request rate.