Hierarchical Network Failure Handling in Clustered Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed computer systems face challenges in efficiently handling network failures within clustered node environments, particularly in determining which nodes to exclude or expel from the cluster to maintain operation and data consistency during network partitions or failures.
Innovation Solution
A hierarchical network failure handling method where a node detects a failure, communicates it upward through the cluster hierarchy, and initiates an expulsion protocol if the failure does not affect higher-level nodes, ensuring that only affected nodes are excluded from the cluster, with higher-level nodes arbitrating to coordinate recovery operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a node detects network failure and initiates expulsion protocol immediately, then the failed node is quickly isolated from the cluster, but this may cause premature expulsion if the failure is actually visible at higher hierarchy levels where coordinated handling is more appropriate
Solution Approach 1:
The cluster is divided into a hierarchical structure with multiple levels (e.g., rack level, pod level, cluster level). Each level independently monitors and handles network failures within its scope. This segmentation allows localized failure handling without requiring global coordination for every event, reducing communication overhead while maintaining reliability through appropriate escalation to higher levels when needed.
Solution Approach 2:
Higher hierarchy levels act as intermediaries that receive failure notifications from lower levels and determine the appropriate response. The intermediary evaluates whether the failure is localized or affects broader scopes, coordinating the expulsion decision across multiple levels to ensure data consistency while avoiding premature or conflicting expulsion actions.
2Speed
If all nodes monitor and can initiate expulsion for the same failure, then failure response is fast, but multiple nodes may simultaneously attempt to expel the same subordinate node causing contention and inconsistent state
Solution Approach 1:
The monitoring and expulsion authority is segmented by hierarchy level. Only the highest level node that detects a failure has the authority to initiate expulsion, preventing multiple nodes from simultaneously expelling the same subordinate. This segmentation maintains fast response time while ensuring consistency through hierarchical arbitration.
Solution Approach 2:
When a node initiates an expulsion protocol, this action is communicated upward through the hierarchy as feedback. Higher levels receive this feedback and adjust their monitoring and decision-making accordingly, preventing duplicate expulsion attempts and ensuring that only one node ultimately expels the failed subordinate, maintaining consistency.
3Reliability
If the cluster uses majority quorum strategy to handle network failures, then the majority subset can continue operation, but the minority subset must be excluded even if they could potentially recover and rejoin quickly
Solution Approach 1:
The expulsion decision is made dynamic rather than static. Instead of permanently excluding minority nodes, the system continuously monitors their status and allows rejoining when they recover. The hierarchical structure enables quick reintegration of recovered nodes at appropriate levels without requiring full cluster quorum re-evaluation, reducing exclusion duration while maintaining operational continuity.
Data Source
AI summary
According to one aspect of the present disclosure, a method and technique for hierarchical network failure handling in a clustered node environment is disclosed. The method includes: detecting a network failure by a node in a cluster, the cluster having plural nodes arranged in a hierarchy, wherein the network failure is associated with a subordinate node in the hierarchy to the detecting node; communicating the network failure from the detecting node to a superior node in the hierarchy; determining whether the network failure affects nodes higher than the detecting node in the hierarchy; and responsive to determining that the network failure does not affect nodes higher than the detecting node in the hierarchy, the detecting node initiating a protocol to expel the subordinate node from the cluster.