Distributed Node Failure Detection Under Central Node Overload
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed systems, central nodes may incorrectly determine the survival state of service nodes due to network delays or high system resource loads, leading to loss or invalidation of survival state information, which results in inaccurate failure detection.
Innovation Solution
A method and apparatus that utilize a state information table to acquire and update survival state information, consider central node status, and adaptively process abnormal service nodes by extending next update times or declaring failure based on network and resource usage data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the central node monitors service nodes by comparing next update time with current system time, then failure detection capability is improved, but false positives occur when the central node is overloaded or network delays prevent timely updates
Solution Approach 1:
The patent changes the monitoring parameters from fixed time-based checks to dynamic checks that consider central node load status and network conditions. When the central node is overloaded or network delays are detected, the system adjusts by extending the next update time threshold, allowing service nodes more time to report updates without being falsely marked as failed.
Solution Approach 2:
The system transitions from a static monitoring approach (fixed time intervals) to a dynamic approach where monitoring parameters adapt based on system conditions. The central node continuously evaluates its own load status and network conditions, and adjusts the failure detection threshold accordingly, making the monitoring mechanism flexible and adaptive to changing system states.
2Reliability
If the central node processes survival state information continuously, then service node monitoring reliability is improved, but system resource consumption increases under high load
Solution Approach 1:
The system implements periodic survival state information collection with dynamic intervals. Instead of continuous monitoring, the central node collects survival state information at periodic intervals, and adjusts these intervals based on system load conditions. When load is high, intervals are extended to reduce processing frequency; when load is low, intervals are shortened to improve detection accuracy.
3Reliability
If the central node extends next update time under high load, then false failure detection is reduced, but service node survival state monitoring timeliness deteriorates
Solution Approach 1:
The next update time threshold is made dynamic rather than fixed. The central node adjusts the next update time based on real-time evaluation of system load and network conditions. When the central node is overloaded, the threshold is extended to prevent false positives; when the system is healthy, the threshold is tightened to improve detection timeliness. This dynamic adjustment resolves the contradiction between accuracy and timeliness.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Example embodiments of the present application provide a method and an apparatus for processing nodes in a distributed system. The nodes include service nodes and a central node, and the method includes: acquiring survival state information of the service nodes; acquiring current system information of the central node; determining, by using the survival state information and the current system information, whether there is an abnormality of the service node; acquiring central state information of the central node if there is an abnormality of the service node; and processing the abnormal service node according to the central state information. The example embodiments of the present application integrate a state of the central node to adaptively process an abnormal service node, thereby reducing wrong determination of a service node state due to problems of the central node and reducing an error probability of the central node.