Cluster Node Failure Detection in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, the time taken to detect and respond to a cluster node failure is excessively long, leading to service stagnation and impaired user experience due to the reliance on inefficient message mechanisms between Monitor nodes and Object-based Storage Devices (OSDs).
Innovation Solution
A method and apparatus that continuously acquire state information from nodes in a cluster to rapidly detect failures, share failure information among nodes, and update the OSDMap, thereby reducing the time to sense node failures and prevent service stagnation. This involves a detection module to identify node failures, a determination module to send failure information to OSDs, and a processing module to update the OSDMap and select a new Monitor, ensuring timely recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the message mechanism between Monitor nodes is used to detect failures, then the system maintains stability through periodic messaging, but the failure detection time becomes excessively long (10 seconds or more)
Solution Approach 1:
The patent implements preliminary action by having Monitor nodes proactively push failure information to OSDs as soon as a failure is detected, rather than waiting for periodic heartbeat messages. The failure information is immediately transmitted to relevant OSDs, which then update the OSDMap and notify other nodes, dramatically reducing the failure detection and response time from 10 seconds to nearly real-time.
2Reliability
If the heartbeat mechanism between OSDs is used to detect failures, then the system ensures thorough detection through mutual heartbeats, but the detection threshold of 1 minute causes excessive service stagnation
Solution Approach 1:
The patent introduces an intermediary mechanism where Monitor nodes act as mediators between failed nodes and the rest of the system. When a Monitor detects a failure, it immediately pushes failure information to OSDs, which then update the OSDMap and notify other OSDs. This intermediary approach eliminates the need for OSDs to wait for mutual heartbeat timeouts, reducing detection time from 1 minute to near-real-time while maintaining detection accuracy.
3Stability of the object's composition
If periodic messaging every 2 seconds is used between Monitor nodes, then the system maintains communication stability, but the 5-message timeout threshold results in 10-second failure detection delay
Solution Approach 1:
The patent implements feedback by having Monitor nodes immediately push failure information back to the system when a failure is detected. Instead of waiting for the periodic message mechanism to timeout, the failed Monitor node's information is actively pushed to OSDs, which then update the OSDMap and notify other nodes. This feedback mechanism maintains communication stability while achieving near-real-time failure detection speed.
Data Source
AI summary
A method and apparatus for processing cluster node failure, a computer device and a readable storage medium. The method includes: circularly acquiring state information of multiple nodes in a cluster, and on the basis of the state information, determining whether a corresponding node fails; in response to failure of the node, sending failure information to multiple OSDs under the node; in response to the multiple OSDs receiving the failure information, according to the failure information, selecting a Monitor to send down information, and setting states of the multiple OSDs to be down; and in response to the Monitor receiving the down information, updating an OSDMap on the basis of the down information, and sending the updated OSDMap to OSDs under other nodes.