Hypervisor Watchdog Recovery for Read-Only Boot Volumes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large-scale event (LSE) models in multi-tenant clouds lack realism and accuracy, leading to prolonged recovery times, often requiring human intervention and causing significant downtime due to issues like power outages and block storage failures.
Innovation Solution
A simulation tool is developed to simulate LSE faults on computing components, allowing for automated recovery process auditing and optimization by launching physical servers, configuring hypervisors, and simulating faults such as power outages or block storage failures, with a watchdog daemon to monitor and recover from read-only boot volumes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional recovery systems are used for LSEs, then recovery can be achieved, but recovery time is prolonged (10 minutes to hours) and requires human intervention
Solution Approach 1:
The system performs preliminary actions by pre-configuring recovery workflows and automating detection mechanisms before failures occur. The watchdog daemon is预先 deployed to monitor system state, and recovery playbooks are prepared in advance, enabling immediate automated response when LSEs occur, thus reducing recovery time from hours to minutes
Solution Approach 2:
The system implements continuous feedback loops through the watchdog daemon that monitors system health and triggers automated recovery actions. Recovery metrics are collected and fed back to improve the system, creating a self-learning mechanism that reduces intervention time and improves reliability over time
2Adaptability or versatility
If existing LSE models are used, then some prediction capability is provided, but the models lack realism and accuracy
Solution Approach 1:
The system creates accurate copies of the actual data center environment by deploying the same hypervisors, operating systems, and workloads in the simulation platform. This copying approach allows realistic LSE simulation and accurate measurement of recovery metrics without affecting production systems
Solution Approach 2:
The system changes key parameters by using real hardware configurations and actual software versions rather than abstract models. By maintaining parameter fidelity between production and simulation environments, the model accuracy and realism are significantly improved
3Adaptability or versatility
If manual intervention is required for recovery, then complex recovery scenarios can be handled, but recovery time increases to hours
Solution Approach 1:
The system enables self-service recovery through automated watchdog daemons that detect failures and execute recovery playbooks without human intervention. The automated system handles complex recovery scenarios by leveraging pre-configured workflows, achieving both high adaptability and fast recovery speed simultaneously
Data Source
AI summary
Techniques discussed herein relate to enabling a hypervisor to self-recover. In particular, a watchdog daemon may be executed at the hypervisor to perform periodic write disk checks of the boot volume associated with the hypervisor. Suppose an attempt to write to disk fails (e.g., an Error Input/Output (EIO) or Error Read Only File System (EROFS) return code is received. In that case, the daemon may determine that the boot volume is in read-only mode, post metrics to one or more logging services to indicate that the daemon has detected a read-only boot volume and reboot the respective hypervisor.