RDMA On-Demand Node Death Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional network heartbeat processing in distributed systems is inefficient in detecting unresponsive nodes, leading to prolonged downtime and latency due to the need for lengthy timeout periods to avoid premature evictions, which can cause significant delays and real-world problems for organizations.
Innovation Solution
Implementing RDMA on-demand validation to quickly confirm the health of nodes by using RDMA messaging, such as RDMA ping or RDMA read requests, which can be processed in microseconds, allowing for faster identification and eviction of dead nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional network heartbeat processing with lengthy timeout periods is used to avoid premature evictions, then node eviction reliability is improved, but system latency and downtime are significantly increased
Solution Approach 1:
The patent applies preliminary action by performing RDMA validation checks proactively before initiating the eviction process. Instead of waiting for timeout periods to expire, the system preemptively validates node responsiveness using RDMA ping or read requests, allowing for immediate detection and eviction of unresponsive nodes without waiting for conventional heartbeat timeouts to elapse.
Solution Approach 2:
The patent substitutes the conventional mechanical heartbeat timing mechanism with RDMA (Remote Direct Memory Access) technology. By replacing the traditional network heartbeat protocol with RDMA-based validation, the system achieves faster node health verification without being constrained by conventional network timeout periods, thereby reducing system latency while maintaining eviction reliability.
2Reliability
If conventional network heartbeat processing is used to detect unresponsive nodes, then node health monitoring is implemented, but detection speed is significantly reduced due to timeout periods
Solution Approach 1:
The patent replaces the conventional network heartbeat mechanism with RDMA-based validation. This substitution enables much faster detection of unresponsive nodes because RDMA operations complete in microseconds compared to the seconds or minutes required by traditional heartbeat timeout mechanisms, while still providing reliable node health monitoring.
Solution Approach 2:
The patent changes the fundamental parameter of detection time by transitioning from conventional network heartbeat timing (measured in seconds or minutes) to RDMA-based validation (measured in microseconds). This parameter change dramatically increases detection speed while maintaining the reliability of node health monitoring through the use of RDMA's reliable communication guarantees.
3Reliability
If lengthy timeout periods are set to confirm node unavailability, then false positive evictions are avoided, but consensus protocol processing is delayed
Solution Approach 1:
The patent applies preliminary action by performing RDMA validation before initiating consensus protocol reprocessing. This preliminary check quickly determines node availability, allowing the consensus protocol to proceed without unnecessary delays while maintaining false positive prevention through RDMA's reliable communication validation.
Solution Approach 2:
The patent substitutes the lengthy timeout-based false positive prevention mechanism with RDMA-based validation. This substitution maintains reliability by using RDMA's guaranteed delivery and acknowledgment features to confirm node unavailability, while dramatically reducing the time required compared to conventional timeout periods, thereby improving consensus protocol processing productivity.
Data Source
AI summary
Described is an improved approach to implement fast detection of node death. Instead of just relying on multiple heart beats to fail in order to determine whether a node is dead, the present approach performs an on demand validation using RDMA to determine whether the node is reachable, where the approach of using RDMA is significantly faster than the heartbeat approach.