API Cluster Rate Limiting with Dynamic Config Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing API Gateway clusters face challenges in implementing global rate limiting due to costly computational overhead and configuration mismatches between envoys and global rate limiters, leading to inefficient request handling and potential inconsistencies.
Innovation Solution
A system utilizing a Global Rate Limiter Service (GRLS) with Customer Resource Definitions (CRDs) to maintain consistency between mapping and rule config files, allowing for dynamic updates and tracking different instance versions, ensuring accurate global rate limiting across API instances without restarting services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If local instance-specific rate limiting is implemented, then each API instance can independently manage its request traffic, but global rate limiting consistency across the API cluster cannot be achieved
Solution Approach 1:
The patent merges local rate limiters across multiple API instances into a unified global rate limiting system. A central rate limiting service aggregates request metrics from all instances and enforces consistent rate limits globally, resolving the contradiction between local independence and global consistency by combining previously separate rate limiting functions into a coordinated system.
Solution Approach 2:
The global rate limiting service performs multiple functions: it collects metrics from various API instances, calculates aggregate request rates, enforces rate limits, and provides configuration management. This multi-functional approach allows a single service to achieve global rate limiting consistency while replacing multiple separate local rate limiting mechanisms.
2Reliability
If a global rate limiting service is implemented, then consistent rate limiting across all API instances can be achieved, but computational overhead and latency increase
Solution Approach 1:
The system performs preliminary actions by pre-calculating rate limit thresholds and maintaining a sliding window of request timestamps. When a rate limit check is needed, the system can quickly determine whether requests exceed limits by comparing against pre-computed metrics rather than analyzing all historical requests, significantly reducing per-request computational overhead and latency.
Solution Approach 2:
The patent uses approximate copying techniques where the global rate limiting service maintains simplified representations of request patterns (such as aggregated counts in sliding windows) rather than processing complete request histories. This copying approach provides sufficiently accurate rate limiting decisions with much lower computational cost, reducing latency while maintaining consistency.
3Adaptability or versatility
If configuration updates are applied to API instances, then rate limiting policies can be updated, but inconsistencies between instances and the global rate limiter may occur
Solution Approach 1:
The global rate limiting service implements feedback mechanisms where configuration changes are propagated to all API instances, and instances report their configuration states back to the service. This feedback loop allows the system to detect and resolve configuration inconsistencies, ensuring all instances remain synchronized with the global rate limiting policy while maintaining adaptability to policy changes.
Solution Approach 2:
API instances automatically apply configuration updates received from the global rate limiting service without requiring manual intervention or service restarts. The instances self-manage their configuration states by comparing received configurations with their current settings and applying updates autonomously, ensuring consistency while maintaining operational continuity.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for implementing global rate limiting of an API cluster capable of dynamically implementing updates without a restart of any instantiation within the API cluster. A local service includes an envoy and a customer resource definition. When an update is received, the customer resource definition identifies changes to be made to a global rate limiting service and dynamically injects those changes into the global rate limiting service. The changes can be instance-specific, with multiple different versions stored for the various instantiations within the cluster. The envoy also extracts and converts header information from a received request into one or more descriptor keys. The global rate limiting service determine global rate limiting based on a set of rules applied to the descriptor keys.