Peer VM Monitoring and Auto-Healing for Sick Node Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual machine (VM) monitoring systems face challenges in accurately identifying sick VMs due to reliance on a few designated nodes, which can become unhealthy, compromising system performance and efficiency, or inefficiently monitoring all nodes, leading to resource utilization issues and slower request processing.
Innovation Solution
Implement a peer VM monitoring system where a selected number of healthy VMs act as leader VMs to periodically check the health of all nodes, storing health information in a data store, and automatically heal sick VMs, ensuring timely identification and recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a small number of designated nodes monitor the health of other nodes, then the monitoring system is simpler to implement, but it is difficult to validate the performance and accuracy of the monitoring system and the designated nodes may become unhealthy themselves
Solution Approach 1:
The system segments the monitoring function by dividing nodes into different roles (leader VMs and follower VMs) rather than having all nodes perform identical monitoring functions. This segmentation allows for a manageable number of leader VMs to perform monitoring while maintaining system reliability through role differentiation and validation capabilities.
2Reliability
If each VM node monitors the health of all other nodes, then the monitoring system has better validation capability, but it consumes more resources and processes requests slower
Solution Approach 1:
Instead of having every node monitor every other node (excessive action), the system implements partial monitoring where only leader VMs monitor follower VMs. This partial action is sufficient to maintain monitoring validation capability while significantly reducing resource consumption and preserving request processing speed.
Solution Approach 2:
The monitoring function is segmented such that only leader VMs perform monitoring tasks while follower VMs focus on processing requests. This segmentation eliminates the need for every node to perform full monitoring, thus maintaining validation capability through the distributed leader VMs while preserving overall system productivity.
3Reliability
If each VM node monitors the health of all other nodes, then the monitoring system provides better validation, but it utilizes more resources increasing system cost
Solution Approach 1:
The system implements partial monitoring action where only leader VMs perform health checks on follower VMs. This partial action provides sufficient monitoring validation to detect sick VMs while consuming far fewer computational resources compared to having every node monitor every other node, thus reducing system cost.
Solution Approach 2:
By segmenting nodes into leader and follower roles, the system concentrates monitoring resource utilization on leader VMs only. This segmentation ensures adequate monitoring validation through the distributed leader VMs while minimizing overall resource consumption and system cost.
Data Source
AI summary
Systems and methods for monitoring health of virtual machines (VMs) include determining a leader virtual machine (VM) count for a group of VM nodes hosted on a plurality of computing devices; selecting a number of the VM nodes of the group to serve as leader VMs for the group, the number of the VM nodes selected corresponding to the leader VM count; and periodically performing a peer VM monitoring process. The peer VM monitoring process includes periodically storing health information for each of the VM nodes of the group in a data store; periodically accessing the health information of each of the VM nodes to identify sick VMs using each of the leader VMs, respectively; and automatically performing a healing process on the sick VMs to improve a performance of the sick VMs.