Dynamic Queue Allocation for Processing Rate Limiting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing rate limiting systems in processing systems suffer from uneven distribution of processing requests among nodes, resource inefficiency due to reliance on tracking nodes and coordinator nodes, and issues like split-brain syndrome, especially when nodes fail.
Innovation Solution
A system that assigns processing requests to queues and allocates these queues dynamically to processing nodes based on current conditions, using a range allocation algorithm to reallocate queue ranges and update node status records, thereby ensuring efficient distribution and limiting processing rates based on calculated backlogs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If shared counters are used to track aggregated request processing rate from all processing nodes, then rate limiting can be enforced at the whole cluster level, but uneven distribution of processing requests among nodes occurs and system resources are consumed for coordination
Solution Approach 1:
The patent divides the centralized rate limiting approach into segmented local rate limiting at each processing node. Each node maintains its own rate limit counter independently rather than sharing a centralized counter, which eliminates the coordination overhead and uneven distribution problems while maintaining effective rate limiting enforcement at each node level
Solution Approach 2:
The patent extracts the rate limiting functionality from the centralized coordination system and places it directly at each processing node. By removing the shared counter mechanism and its associated coordination protocols, the system eliminates the resource consumption and distribution inefficiencies while preserving rate limiting capability
2Ease of operation
If tracking nodes and coordinator nodes are designated to allocate processing rates among processing nodes, then processing rate allocation can be managed, but additional system resources are consumed that could be used for processing requests
Solution Approach 1:
The patent implements self-service rate limiting where each processing node autonomously manages its own rate limit counter without requiring tracking nodes or coordinator nodes. Each node independently tracks its own request processing rate and enforces rate limits locally, eliminating the need for dedicated coordination infrastructure and freeing up system resources for actual request processing
3Reliability
If token bucket algorithm or leaky bucket algorithm is used to track inbound requests, then rate limits can be enforced, but requests are rejected when tokens or counters exceed a predetermined limit causing uneven distribution
Solution Approach 1:
The patent segments the rate limiting state into local counters at each processing node rather than using global token buckets or leaky buckets. Each node maintains independent rate limit state, allowing requests to be processed efficiently without the uneven distribution and rejection issues that arise from centralized token/counter mechanisms
Data Source
AI summary
Systems and methods including one or more processors and one or more non-transitory storage devices storing computing instructions configured to run on the one or more processors and perform acts of receiving one or more processing requests; assigning each respective processing request of the one or more processing requests to a respective queue of one or more queues; assigning each respective queue of the one or more queues to a respective processing node of one or more processing nodes; calculating a respective processing request backlog for each respective processing node of the one or more processing nodes; and limiting a processing rate of the respective processing node for processing requests of the one or more processing requests of the respective queue based on the respective processing request backlog for the respective processing node. Other embodiments are disclosed herein.