Virtual Machine Storage Failure Recovery via Fault Domain Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hardware failures, particularly storage and network outages, significantly impact virtualized infrastructure, leading to application downtime and reduced confidence in virtualization, as the number of virtual machines and datastores per host increases, thereby limiting consolidation ratios and capital expenditure benefits.
Innovation Solution
The system monitors virtual machines for storage access failures and applies automated recovery methods, such as terminating and restarting virtual machines on healthy hosts, to mitigate the impact of Permanent Device Loss (PDL) and transient All Paths Down (APD) failures, using a heuristic approach and fault domain management to ensure continuous service.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of virtual machines and datastores per host is increased to improve consolidation ratios and capital expenditure benefits, then productivity and cost efficiency are improved, but reliability deteriorates because hardware failures impact more virtual machines
Solution Approach 1:
The system segments the virtualized environment into fault domains, isolating failures to specific segments. When a storage device fails, only virtual machines in that specific fault domain are affected, while other segments continue operating normally. This segmentation limits the propagation of failures and maintains overall system reliability despite high consolidation ratios.
Solution Approach 2:
The system dynamically changes operational parameters of virtual machines based on storage availability. When storage connectivity is lost, the system modifies the state of affected virtual machines (suspending or migrating them) to prevent complete system failure. This parameter adjustment allows the system to maintain service availability even with high consolidation where hardware failures are more impactful.
2Reliability
If automated recovery methods are implemented to improve reliability and reduce downtime, then service availability is improved, but device complexity increases due to monitoring and remediation systems
Solution Approach 1:
The system implements self-service automation where virtual machines automatically detect storage connectivity issues and trigger their own recovery processes. The monitoring system detects failures and automatically suspends or migrates affected virtual machines without requiring manual intervention. This self-service approach improves reliability through rapid automated response while managing complexity by using standardized recovery procedures that virtual machines can execute independently.
Solution Approach 2:
The system establishes continuous feedback loops between storage devices, virtual machines, and the monitoring system. Storage connectivity status is continuously monitored and fed back to the virtual machine management system, which automatically adjusts virtual machine operations based on this feedback. This feedback mechanism enables reliable automated recovery while keeping complexity manageable through rule-based response protocols rather than complex decision-making algorithms.
3Device complexity
If manual monitoring and intervention are used to handle storage failures, then device complexity is reduced, but loss of time increases due to delayed detection and response to failures
Solution Approach 1:
The system performs preliminary actions by pre-configuring recovery protocols and automatically suspending virtual machines before complete failure occurs. When storage connectivity issues are detected, the system proactively migrates or suspends affected virtual machines to prevent data loss or service interruption. This preliminary action approach minimizes downtime by addressing failures immediately upon detection, while the automated nature of these pre-configured responses keeps complexity manageable.
Data Source
AI summary
A system for monitoring a virtual machine executed on a host. The system includes a processor that receives an indication that a failure caused a storage device to be inaccessible to the virtual machine, the inaccessible storage device impacting an ability of the virtual machine to provide service, and applies a remedy to restore access to the storage device based on a type of the failure.