Pooled Memory Heartbeat for Host Status Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In high availability clusters, determining whether a host is crashed or merely isolated due to network disconnection is challenging, leading to unnecessary virtual machine restarts and resource consumption, as existing methods struggle to accurately differentiate between operational and failed hosts.
Innovation Solution
Implementing a pooled memory heartbeat system where each host creates and updates a heartbeat file in shared memory, allowing agents to monitor the status by checking the lock status of the file, thereby distinguishing between active and inactive hosts and reducing incorrect failure identifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If network heartbeat monitoring is used to detect host status, then network connectivity can be monitored, but false positives occur when hosts are network isolated but still operational
Solution Approach 1:
The patent introduces pooled memory as an intermediary communication channel between hosts and the HA agent. Instead of relying solely on network heartbeats, the system uses memory-based heartbeat files that hosts can write to and agents can read from, providing a secondary verification path that remains accessible even when network connectivity is lost. This mediator resolves the contradiction by providing reliable host status information independent of network conditions.
2Reliability
If virtual machines are restarted on alternative hosts when network isolation is detected, then high availability is maintained, but unnecessary restarts consume cluster resources
Solution Approach 1:
The system implements feedback by continuously monitoring pooled memory heartbeat files and using this information to verify host status before triggering VM restarts. The HA agent checks whether the heartbeat file has been recently updated by the isolated host, providing feedback that distinguishes between truly failed hosts and those merely experiencing network isolation. This feedback mechanism prevents unnecessary VM restarts and conserves cluster resources while maintaining high availability for genuinely failed hosts.
3Reliability
If shared data store heartbeat is used to determine host failure, then storage connectivity can be monitored, but hosts may be incorrectly marked as failed when disconnected from shared data store
Solution Approach 1:
The patent segments the heartbeat monitoring function into multiple independent components: network heartbeat monitoring, pooled memory heartbeat monitoring, and shared data store heartbeat monitoring. Each component monitors a different aspect of host connectivity independently. By segmenting these monitoring functions, the system can evaluate each separately and avoid false failures caused by any single connectivity issue, thereby improving measurement precision of actual host operational status.
Data Source
AI summary
Examples provide a pooled memory heartbeat for virtual machine hosts. A virtual controller creates a pooled memory heartbeat file system in a shared memory partition of a pooled memory. An agent running on each host in a plurality of virtual machine hosts updates a heartbeat file at an update time interval to lock the heartbeat file. The lock indicates the heartbeat status for a given host is active. A master agent accesses the shared memory partition to check the heartbeat status of each host in the pooled memory file system. The heartbeat status is used to determine whether a host that has lost pooled memory access, is network isolated, or failed. If the pooled memory heartbeat status for a given host indicates the host is a failed host, the set of virtual machines running on the given host are respawned on another healthier host.