Cluster Node Fault Isolation via Topology Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In Akka cluster networks, existing fault detection mechanisms often incorrectly remove healthy nodes, leading to high service unavailability and increased overheads during fault rectification due to the inability to accurately distinguish between link and node faults, resulting in inefficient cluster management and high costs.
Innovation Solution
An intra-cluster node troubleshooting method that utilizes fault detection topology information and fault indication messages to accurately identify and isolate faulty nodes and links, allowing for the retention of available nodes and minimizing service unavailability by determining a working sub-cluster with the maximum number of nodes, thereby reducing the probability of cluster restart and associated costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Akka fault detection mechanism is used to detect link faults or packet loss faults, then connection failure between nodes is detected, but normal nodes are incorrectly removed with high probability
Solution Approach 1:
The patent segments the fault detection process into two distinct phases: connection detection (using Akka's existing mechanism) and fault type discrimination (using the new ping mechanism). This segmentation allows the system to first identify that a fault exists, then separately determine whether it is a link fault or node fault, thereby avoiding incorrect removal of normal nodes while maintaining reliable fault detection
Solution Approach 2:
The patent introduces a ping mechanism as an intermediary step between connection detection and fault decision-making. When connection failure is detected, the system uses ping to intermediary check node reachability through other nodes in the cluster. This intermediary mechanism provides additional information to accurately distinguish between link faults and node faults, preventing premature removal of healthy nodes
2Reliability
If delegated nodes are used to ping the detected node when heartbeat is not detected, then node reachability can be confirmed, but time spent on fault detection and data synchronization volume increase
Solution Approach 1:
The patent applies partial action by selectively triggering the ping mechanism only when connection failure is detected, rather than continuously pinging all nodes. This partial application of the detection mechanism reduces overall time consumption and data synchronization overhead while maintaining reliable fault detection accuracy when needed
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Embodiments of this application relate to an intra-cluster node troubleshooting method and device. The method includes: obtaining fault detection topology information of a cluster, where the fault detection topology information includes a fault detection relationship between all nodes in the cluster; obtaining a fault indication message, where the fault indication message is used to indicate unreachability from a detection node to a detected node; determining a sub-cluster of the cluster based on the fault detection topology information and the fault indication message, where nodes that belong to different sub-clusters are unreachable to each other; and determining a working cluster based on the sub-cluster of the cluster. According to the embodiments of this application, available nodes in the cluster can be retained to a maximum extent at relatively low costs. In this way, a quantity of available nodes in the cluster is increased, high availability is ensured, a probability of service unavailability caused by cluster restart is reduced, and overheads of fault rectification and service migration are reduced.