NIC-Scheduled Heartbeat Agents for Deterministic Node Failure Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing heartbeat mechanisms in distributed systems suffer from high latency and inconsistency due to OS scheduling, leading to delayed node failure detection and impacting real-time applications.
Innovation Solution
Implementing a heartbeat agent within each node close to the transmission media, utilizing a time-based packet scheduling feature of network interfaces to transmit probe and response packets, thereby avoiding OS scheduling and ensuring consistent round-trip times for deterministic failure detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If OS scheduling is used to transmit heartbeat packets, then the system is easier to operate and integrate with existing OS structures, but the latency increases and consistency deteriorates
Solution Approach 1:
The patent extracts the heartbeat packet transmission function from the OS scheduling layer and implements it directly at the network interface card (NIC) level. The NIC autonomously schedules and transmits heartbeat packets based on pre-configured time intervals, eliminating the need for OS intervention and reducing latency while maintaining deterministic timing.
Solution Approach 2:
The system pre-configures the NIC with heartbeat transmission parameters including time intervals, destination addresses, and packet formats before runtime. This preliminary configuration enables the NIC to autonomously execute heartbeat transmissions without real-time OS processing, achieving low-latency and consistent timing.
2Reliability
If multiple heartbeat intervals are waited for before detecting failure, then false positives are reduced, but the failure detection time increases
Solution Approach 1:
The patent replaces the traditional software-based heartbeat processing mechanism with a hardware-accelerated approach at the NIC level. The NIC handles packet transmission and timing-critical operations in hardware, providing deterministic and consistent round-trip times that enable more reliable single-interval failure detection without false positives.
Solution Approach 2:
The system changes the timing parameters of heartbeat transmissions by using fixed, pre-configured intervals enforced by the NIC hardware rather than OS-scheduled variable intervals. This creates predictable and consistent round-trip times, allowing the system to reliably detect failures within a single heartbeat interval with high confidence.
3Loss of time
If the heartbeat module is located far from transmission media, then integration with OS is easier, but the processing overhead and latency increase
Solution Approach 1:
The patent extracts the heartbeat transmission and reception functionality from the upper-layer OS networking stack and implements it directly within the NIC firmware or hardware logic. This proximity to the transmission media eliminates multiple data copies and processing steps, significantly reducing packet processing time and latency.
Solution Approach 2:
The NIC acts as an intermediary between the OS and the transmission media, handling heartbeat packet generation, scheduling, and transmission autonomously. This intermediate layer at the NIC level reduces the processing burden on the OS while maintaining system integration, achieving low latency without excessive complexity.
Data Source
AI summary
Embodiments described herein are generally directed to a flexible mechanism for performing low latency and deterministic node failure detection. In an example, an agent, running on a monitor node of a distributed system, receives from a process, running within a user space of an operating system (OS) of the monitor node, a request to send a probe to a monitored node of the distributed system. The agent is interposed between a networking stack of a kernel of the OS and a transmission media coupling the nodes in communication. The agent causes the probe to be transmitted to the monitored node via the transmission media at a time specified by the request utilizing a time-based packet scheduling feature of a network interface associated with the monitor node. When a time period elapses prior to receipt of a response to the probe, the agent notifies the process of a failure relating to the monitored node.