Node Health Verification via Peer Querying for Workload Recovery

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing container orchestration systems face challenges in accurately determining the health status of worker nodes when disconnected from the control plane, leading to potential false shut downs due to transient resource unavailability or network splits, which can result in unnecessary downtime and inefficient workload recovery.

Innovation Solution

A method where a node periodically transmits polling signals to the control plane to assess its health status and sends heartbeat signals to a watchdog device only when healthy, and if communication is lost for a threshold, it queries a subset of peer nodes to verify its status, preventing false shut downs by obtaining a more accurate assessment of its health.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If a worker node periodically transmits polling signals to the control plane to assess health status, then the accuracy of health determination is improved, but the network communication overhead and system complexity increase

Engineering Contradiction:

Improvehealth status determination accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The worker node transmits polling signals to the control plane at periodic intervals to assess its health status. This periodic action allows the system to maintain accurate health determination without requiring continuous communication, thereby balancing measurement precision with reduced network overhead and system complexity.

Inventive Principle:

Principle #19Periodic action

2Reliability

If a worker node shuts down when disconnected from the control plane, then the system reliability is improved by removing unhealthy nodes, but false shut downs occur due to transient resource unavailability or network splits

Engineering Contradiction:

Improvesystem reliabilityVSAvoidhealth status accuracy

Core Design Contradiction:

ReliabilityVSMeasurement precision

Solution Approach 1:

Before shutting down, the worker node performs preliminary actions by querying a subset of peer nodes to verify its health status. This preliminary verification step prevents false shut downs caused by transient issues while maintaining system reliability by still removing genuinely unhealthy nodes.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The worker node implements a feedback mechanism by receiving responses from peer nodes about its health status. This feedback loop allows the node to make informed decisions about shutdown, improving health status accuracy while maintaining system reliability through verified node removal.

Inventive Principle:

Principle #23Feedback

3Measurement precision

If a worker node queries peer nodes to verify health status before shutdown, then false shut downs are prevented, but the time required for health assessment increases

Engineering Contradiction:

Improvehealth status accuracyVSAvoidhealth assessment time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The worker node queries only a subset of peer nodes rather than all nodes in the cluster. This partial action approach maintains health status accuracy by obtaining sufficient verification information while reducing the time required for health assessment compared to querying the entire peer set.

Inventive Principle:

Principle #16Partial or excessive action

4Reliability

If the control plane cannot distinguish between network splits and worker node failures, then unnecessary shut downs are prevented, but the ability to rapidly respond to actual failures is reduced

Engineering Contradiction:

Improvefalse shutdown preventionVSAvoidworkload recovery speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The health verification process is segmented into multiple independent steps: initial disconnection detection, peer node querying, and conditional shutdown execution. This segmentation allows the system to maintain reliability by preventing false shut downs while preserving productivity through rapid workload recovery once genuine failures are confirmed through the segmented verification process.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11397632B2Safely recovering workloads within a finite timeframe from unhealthy cluster nodes

Publication Date: 2022.07.26 RED HAT LLC

AI summary

Systems and methods for managing nodes that have lost contact with a control plane are disclosed. A first node among a plurality of nodes may use a polling process to monitor for a polling signal response from a control plane at periodic intervals, wherein the polling signal response indicates a health status of the first node as healthy. In response to not receiving the polling signal response for a threshold number of consecutive intervals, the first node may identify a subset of peer nodes among the plurality of nodes and instruct each peer node in the subset to inquire as to the health status of the first node from the control plane. The first node may perform an action based on a response received from each of the subset of peer nodes after inquiring as to the health status of the first node from the control plane.