Node Crash Detection via Multi-Probe Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing systems rely on unreliable methods to detect node crashes, leading to false positives and unnecessary remediation processes, as they cannot deterministically confirm a node crash based on loss of communication, which wastes resources and is inefficient.
Innovation Solution
Implementing a system that uses multiple probes, including an API call to a crash kernel and a Serial Over LAN (SOL) probe of the console, to confirm a node crash before initiating remediation, ensuring accurate detection and minimizing false positives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple probes are used to confirm node crash, then detection reliability is improved, but device complexity increases
Solution Approach 1:
The detection system is segmented into multiple independent probe mechanisms: API probes that communicate with the crash kernel, SOL probes that access the serial console, and orchestration status monitoring. Each probe operates independently and contributes specific detection capabilities, allowing the system to achieve high reliability through diversified detection approaches rather than relying on a single complex detection mechanism.
2Reliability
If multiple probes are used to confirm node crash, then false positives are reduced, but loss of time increases
Solution Approach 1:
The system performs preliminary actions by continuously monitoring orchestration status and maintaining readiness of multiple probe mechanisms before actual crashes occur. When a node becomes unavailable, the API probe and SOL probe are immediately triggered simultaneously rather than sequentially, and the crash kernel is pre-configured to respond to these probes. This preliminary preparation minimizes detection time while ensuring accurate crash confirmation through multiple verification methods.
Data Source
AI summary
A computer-implemented method includes: detecting, by one or more processors, an indication that suggests a node has crashed, wherein the node is included in a distributed computing environment; in response to the detecting, confirming by the one or more processors whether the node has crashed by performing a set of probes on the node; and in response to the confirming that the node has crashed, initiating by the one or more processors a remediation of the node.