Fair Decentralized Throttling in Distributed Cloud Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing systems face challenges in managing API requests during recovery workflows, as unthrottled API calls can overwhelm the infrastructure, leading to service disruptions and fairness issues across tenants, especially in hybrid cloud environments.
Innovation Solution
Implementing a central API gateway that manages queues for synchronous and asynchronous requests, setting tenant-specific limits to prevent overload and ensure fair resource allocation across tenants, thereby throttling API calls to prevent cascading failures and maintain system availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If API requests are throttled to prevent system overload, then system stability is improved, but request processing speed deteriorates
Solution Approach 1:
The patent implements dynamic throttling where the API gateway adjusts request rate limits based on real-time system conditions. The gateway monitors system load, queue depths, and component health status, then dynamically modifies throttling parameters to balance system stability with request processing speed. This allows the system to be more permissive during low-load conditions and more restrictive during high-load conditions.
Solution Approach 2:
The patent changes throttling parameters based on system state. Different rate limits, queue sizes, and rejection thresholds are applied depending on current system conditions such as recovery workflow status, component availability, and overall load. This enables the system to optimize between stability and speed by adjusting parameters rather than using fixed thresholds.
2Adaptability or versatility
If tenant-specific limits are enforced to ensure fair resource allocation, then fairness across tenants is improved, but system complexity deteriorates
Solution Approach 1:
The patent segments the API gateway functionality into distinct modules: tenant identification module, quota tracking module, request prioritization module, and queue management module. Each module handles a specific aspect of multi-tenant management, making the overall complex system manageable through clear separation of concerns. This segmentation enables fair resource allocation while keeping individual component complexity low.
Solution Approach 2:
The API gateway implements a universal throttling framework that serves all tenants through a single standardized mechanism. Rather than implementing separate throttling systems for each tenant, the gateway provides universal rate limiting, queue management, and priority handling that automatically applies to all tenants based on their configured quotas and SLAs. This reduces overall system complexity while maintaining fairness.
3Loss of time
If synchronous requests are processed immediately to maintain responsiveness, then response time is improved, but system overload increases
Solution Approach 1:
The patent applies partial action by processing only a subset of synchronous requests immediately based on available system capacity. The API gateway evaluates current load conditions and selectively processes requests that can be handled without causing overload, while queuing or rejecting others. This partial processing approach maintains responsiveness for critical requests while preventing system overload.
Solution Approach 2:
The patent introduces an intermediary queueing mechanism between request reception and processing. The API gateway acts as an intermediary that buffers synchronous requests, allowing immediate acknowledgment to clients while deferring actual processing to when system capacity is available. This intermediary layer decouples request acceptance from request processing, reducing immediate system load while maintaining apparent responsiveness.
4Reliability
If asynchronous requests are queued for later processing to reduce immediate load, then system stability is improved, but processing delay increases
Solution Approach 1:
The patent implements continuous processing of asynchronous requests from the queue rather than batch processing. The API gateway continuously monitors queue depth and system capacity, processing requests as soon as capacity becomes available. This continuous action minimizes waiting time while maintaining system stability by processing requests at a sustainable rate rather than allowing large accumulations that would cause delays.
Solution Approach 2:
The patent employs feedback mechanisms where the API gateway continuously monitors queue depth, processing rate, and system load conditions. Based on this feedback, the gateway dynamically adjusts processing priorities and rates. When system capacity increases, the gateway accelerates queue processing to reduce delays. When load increases, it moderates processing to maintain stability, creating a feedback loop that balances both objectives.
Data Source
AI summary
A cloud availability manager configured to execute a recovery workflow that fails over one or more virtual machines (VMs) to and from a cloud computing system. In doing so, the cloud availability manager typically performs multiple operations for each VMs. The operations involve making several application programming interface (API) calls to component APIs of management components within the cloud computing system. To avoid bringing down the entire cloud infrastructure, the cloud availability manager throttles the API calls to other components while executing a recovery workflow. The throttling spans multiple instances (nodes) of the cloud availability manager and involves cooperation from the other management components to ensure the throttling is fair across all tenants of the cloud computing system.