Node Death Detection via Subnet Manager Query

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed computing environments face delays in detecting node death, leading to brownouts due to the reliance on ping-based health detection methods, which can result in prolonged network downtime and resource unavailability.

Innovation Solution

Implementing a node death detection mechanism that combines a heartbeat mechanism with a query to the subnet manager in an InfiniBand network, allowing for quicker and more reliable determination of node status by verifying the presence of port identifiers in the subnet manager's port table.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If ping-based health detection is used with periodic intervals and miss count thresholds, then false positives are reduced, but node death detection time increases to 30+ seconds

Engineering Contradiction:

Improveaccuracy of node death detectionVSAvoidnode death detection time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by querying the subnet manager for port identifier presence immediately when a node is suspected of failure, rather than waiting for the full ping miss count threshold to be reached. This preliminary check provides early indication of node status, reducing the overall detection time while maintaining reliability through the combination with ping-based detection.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The subnet manager acts as an intermediary that provides additional verification information about node status through port identifier queries. This intermediary mechanism complements the ping-based detection by providing a different perspective on node liveness, allowing for faster and more reliable detection without increasing false positives.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Loss of time

If aggressive ping intervals and low miss count thresholds are used, then node death detection time is reduced, but false positives increase

Engineering Contradiction:

Improvenode death detection timeVSAvoidaccuracy of node death detection

Core Design Contradiction:

Loss of timeVSReliability

Solution Approach 1:

The subnet manager query serves as an intermediary verification step that reduces false positives. By checking port identifier presence in the subnet manager, the system gains additional confirmation of node status before declaring death, allowing for more aggressive ping timing without sacrificing reliability.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent substitutes part of the mechanical ping-acknowledgement system with a subnet manager query mechanism. This substitution provides a different method of verifying node status that is less prone to false positives, enabling more aggressive detection timing.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If the system waits for multiple consecutive ping failures before declaring node death, then false positives are minimized, but resource availability is reduced during brownout period

Engineering Contradiction:

Improveaccuracy of node death detectionVSAvoidresource availability

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system takes preliminary action by querying the subnet manager early in the failure detection process. This preliminary check provides faster indication of actual node failure, allowing the system to begin resource reconfiguration earlier while maintaining reliability through the combination with ping-based detection.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The subnet manager query provides feedback about port identifier presence that complements the ping feedback mechanism. This additional feedback channel allows for faster and more accurate determination of node status, improving both reliability and productivity by reducing the brownout period.

Inventive Principle:

Principle #23Feedback

4Device complexity

If the system uses a single detection mechanism, then device complexity is reduced, but detection reliability and speed are compromised

Engineering Contradiction:

Improvedetection mechanism complexityVSAvoidnode death detection accuracy

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The patent merges two detection mechanisms: ping-based health detection and subnet manager port identifier verification. This combination leverages the strengths of both approaches, providing faster and more reliable node death detection while maintaining manageable system complexity through integrated implementation.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS9135097B2Node death detection by querying

Publication Date: 2015.09.15 ORACLE INT CORP

AI summary

Systems, methods, and other embodiments associated with detecting a node death in a clustered distributed system are described. In one embodiment, a method includes transmitting a ping message to a peer node in the network. If a reply to the ping message is not received from the peer node, a query is sent to table of port identifiers that lists ports in the cluster. In one embodiment, the query includes a port identifier associated with the peer node. The peer node is declared as inactive/dead when the query fails to locate a match in the table for the port identifier. When the query locates a match in the table for the port identifier, another ping message is periodically transmitted to the peer node.