Client-Side Token Bucket for SaaS API Throttling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software as a Service (SaaS) vendors impose throughput penalties on user groups due to excessive API events, which are not transparently managed, leading to inefficiencies and service disruptions, as existing approaches fail to effectively predict and adapt to opaque server limits, impacting service level agreements (SLAs) and system throughput.
Innovation Solution
Implementing dynamic token bucket adjusting techniques on the client-side to infer server load conditions and dynamically set throttle limits, thereby proactively managing API call rates to avoid triggering throughput penalties, ensuring efficient resource utilization and maintaining SLA guarantees.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dynamic token bucket adjusting is implemented to proactively manage API call rates, then throughput penalties are avoided and SLA guarantees are maintained, but system complexity increases due to client-side inference and dynamic adjustment mechanisms
Solution Approach 1:
The system performs preliminary actions by inferring server load conditions and dynamically adjusting API call rates before throughput penalties are triggered. The client-side agent proactively monitors and adapts to server conditions, preventing service disruptions rather than reacting to them after occurrence.
Solution Approach 2:
The system implements feedback mechanisms where the client-side agent continuously monitors server responses and inferred load conditions, then dynamically adjusts the token bucket parameters accordingly. This closed-loop control ensures SLA compliance while adapting to changing server states.
2Reliability
If API call rates are dynamically adjusted based on inferred server load, then service disruptions are prevented, but latency increases for power users due to throttling
Solution Approach 1:
The system applies different quality levels to different users by identifying power users and applying selective throttling. Non-power users experience minimal impact with reduced latency, while power users are subject to more aggressive rate limiting when server load is high, ensuring service continuity for the majority.
Solution Approach 2:
The system applies partial throttling selectively to power users rather than uniformly limiting all API calls. By identifying and targeting only the excessive API calls from power users, the system maintains service continuity while minimizing the impact on overall system performance and user experience.
3Productivity
If opaque server limits are not transparently managed, then vendor control is maintained, but efficiency decreases due to unpredictable throughput penalties
Solution Approach 1:
The client-side agent acts as an intermediary between the client application and the SaaS server, inferring server load conditions and dynamically adjusting API call rates. This intermediary layer translates opaque server limits into actionable insights, enabling efficient rate management without requiring direct transparency from the vendor.
Solution Approach 2:
The system enables self-service by allowing the client-side agent to autonomously infer server conditions and adjust API call rates without requiring vendor-provided transparency. The agent independently monitors responses, infers load conditions, and adapts rate limits, empowering the client to manage efficiency despite opaque server limits.
Data Source
AI summary
A method and system for reducing triggering of throughput penalties imposed on a group of users by a software-as-a-service (SaaS) server due to Application Programming Interface (API) calls exceeding limits of the SaaS server. The approaches include intercepting requests to the SaaS server from a user group and monitoring a rate of API calls the API calls forwarded to the SaaS server, identifying one or more power users based on a notification threshold value for the user group, and managing the rate of the API calls for the requests submitted by the identified power users of the user group in accordance with an API call throttle limit, thus remediating triggering of the throughput penalty.