Request Cost Index for API Throttling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing API request throttling techniques struggle to balance scalability and precision, often failing to accurately measure computing resource utilization, leading to unfair usage of finite resources and frustration for users.

Innovation Solution

The implementation of a request cost index that calculates the amount of computing resources used by API requests, adjusting a state variable to throttle subsequent requests based on the actual resource utilization, rather than just the request rate, ensuring fair use and high precision.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional request rate limiting is used to prevent capacity overload, then system stability is improved, but measurement precision of resource utilization deteriorates

Engineering Contradiction:

Improvesystem stabilityVSAvoidresource utilization measurement

Core Design Contradiction:

ReliabilityVSMeasurement precision

Solution Approach 1:

The patent changes the parameter used for rate limiting from simple request count to a composite metric (request cost index) that incorporates multiple dimensions including compute time, memory usage, storage operations, and network bandwidth. This allows the system to maintain stability while achieving precise measurement of actual resource consumption patterns.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the mechanical counting mechanism of traditional rate limiters with an instrumented measurement system that uses telemetry data, metrics collection, and computational analysis to determine actual resource utilization. This substitution enables precise measurement by replacing simple increment operations with comprehensive resource monitoring.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If request throttling is applied to ensure fair resource usage, then resource fairness is improved, but productivity of legitimate requests deteriorates

Engineering Contradiction:

Improveresource fairnessVSAvoidrequest processing throughput

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent applies differentiated rate limiting policies to different users, APIs, and resource types based on their specific consumption patterns and fairness requirements. Instead of uniform throttling, the system adjusts limits locally for each context, ensuring fairness while minimizing impact on legitimate high-value requests.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent implements dynamic rate limiting where thresholds and penalties are adjusted in real-time based on current system load, user history, and resource availability. This dynamic approach allows the system to be more permissive during low-load periods and more restrictive during peak times, maintaining fairness without unnecessarily blocking productive requests.

Inventive Principle:

Principle #15Dynamics

3Measurement precision

If comprehensive resource monitoring is implemented to achieve precise measurement, then measurement precision is improved, but device complexity deteriorates

Engineering Contradiction:

Improveresource utilization measurementVSAvoidsystem architecture complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the resource monitoring system into independent modules that track different resource types (compute, memory, storage, network) separately. Each module collects and reports metrics independently, which simplifies the overall architecture by avoiding the need for a single complex monolithic monitoring system while achieving comprehensive measurement precision.

Inventive Principle:

Principle #1Segmentation

4Measurement precision

If request cost index calculation is performed for each operation, then measurement precision is improved, but use of energy deteriorates

Engineering Contradiction:

Improveresource utilization measurementVSAvoidcomputational energy consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent implements partial measurement by calculating the request cost index selectively based on operation type, user profile, and current system state. Not all operations require full comprehensive measurement - the system applies appropriate levels of measurement precision to each case, reducing unnecessary computational energy consumption while maintaining adequate measurement precision where needed.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS12182114B1Request cost index for throttling requests to execute operations in a multi-tenant provider network

Publication Date: 2024.12.31 AMAZON TECH INC

AI summary

Techniques for calculating and using a request cost index for throttling application programming interface (API) requests to execute operations in a provider network. The techniques encompass the step receiving a request to execute an operation at an API service in the provider network. Further steps include determining to execute the operation based on a request rate limiting algorithm, executing the operation to yield an operation result, and sending the operation result. Additional steps include calculating a request cost index that reflects an amount of computing resources utilized by executing the operation, determining an adjustment amount for a state variable of the request rate limiting algorithm based on the calculated request cost index, and adjusting (e.g., lowering) the state variable by the adjustment amount. Other (e.g., subsequent) requests to execute queries received at the API service that are metered by the state variable can be throttled by the API service.