Kubernetes Pod Startup Ordering to Prevent Thundering Herds

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In Kubernetes-based systems, resource starvation and service throttling occur due to limited computing resources, leading to thundering herd problems during operational changes like power cycling and software upgrades, causing performance degradation and potential data loss.

Innovation Solution

Implementing a pluggable mechanism that controls pod and container creation through plugins, ensuring orderly resource allocation and visibility into delays to prevent overloading and throttling.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multiple pods and containers are created simultaneously during operational changes, then service recovery and system availability are improved, but resource starvation and service throttling occur due to limited computing resources

Engineering Contradiction:

Improveservice availabilityVSAvoidresource starvation

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

The patent implements rate limiting that controls pod and container creation by introducing periodic delays and throttling mechanisms. The rate limiter monitors resource usage and regulates the frequency of pod creation operations, preventing simultaneous mass creation that causes resource starvation while still enabling systematic service recovery.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The patent applies preliminary action by pre-configuring rate limiting policies and resource quotas before operational changes occur. The system establishes creation rates and resource allocation limits in advance, so when pod creation is needed during failures or upgrades, the rate limiter is already in place to guide resource allocation and prevent throttling.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If pod creation rate is increased to recover services faster, then service availability improves, but thundering herd problems occur causing performance degradation

Engineering Contradiction:

Improveservice recovery speedVSAvoidthundering herd problem

Core Design Contradiction:

ProductivityVSObject-generated harmful factors

Solution Approach 1:

The rate limiter introduces periodic control by regulating pod creation rates and implementing exponential backoff strategies. Instead of allowing uncontrolled simultaneous creation, the system enforces time-based rate limits that space out pod creation operations, preventing the thundering herd effect while maintaining steady service recovery progress.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The patent implements feedback mechanisms where the rate limiter continuously monitors resource usage, pod creation rates, and system health metrics. Based on this feedback, the system dynamically adjusts creation rates to prevent thundering herd problems, slowing down when resources are constrained and allowing faster creation when capacity is available.

Inventive Principle:

Principle #23Feedback

3Stability of the object's composition

If resource allocation is strictly controlled to prevent starvation, then resource stability is improved, but service creation delays increase

Engineering Contradiction:

Improveresource stabilityVSAvoidservice creation delay

Core Design Contradiction:

Stability of the object's compositionVSLoss of time

Solution Approach 1:

The rate limiting system implements dynamic resource allocation that adjusts pod creation rates based on real-time resource availability and system conditions. Rather than applying fixed strict limits, the system adapts creation rates dynamically, allowing faster creation when resources are abundant and implementing delays when resources are constrained, thus balancing stability with creation speed.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent changes parameters such as rate limits, quotas, and creation thresholds based on system state. The rate limiter modifies resource allocation parameters dynamically, adjusting the stringency of resource control based on current workload, resource capacity, and service recovery needs, thereby optimizing both stability and creation timing.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20260050490A1Method and system for preventing resource starvation and service throttling in a kubernetes-based system

Publication Date: 2026.02.19 CISCO TECHNOLOGY INC

AI summary

Techniques and mechanisms for preventing resource starvation and service throttling during container orchestration system operations are provided. In a Kubernetes-based container orchestration system, pluggable mechanisms are applied to pod and/or associated container creation and operation at the direction of a cluster controller to prevent resource overloading and service throttling during thundering herd problems encountered by components of a Kubernetes cluster. A special-purpose plugin may be provided that causes creation of a pod and/or associated container according to a prescribed order to prevent caching overload and resulting thundering herd problems during pod and/or associated container creation and startup.