Per-VM HA State Persistence in Distributed Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The integration of high availability (HA) with hypervisor-converged object-based (HC/OB) storage systems poses challenges due to the lack of coarse storage containers, leading to issues in HA protection state persistence and VM failover/restart workflows, particularly in network partition scenarios where accessibility of storage objects is asymmetric and conventional coarse-grained storage checks are insufficient.
Innovation Solution
The solution involves persisting HA protection state for VMs on a per-VM basis within the logical storage pool of the distributed object-based storage system, modifying the failover/restart workflow to account for potential VM inaccessibility, and enhancing workflows to operate effectively in network partition scenarios by using namespace objects and VSAN layer APIs for metadata management and accessibility checks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If HA protection state is persisted in a centralized file on dedicated shared storage, then HA management is simplified and VM failover is reliable, but the system cannot leverage HC/OB storage architecture where storage is distributed across host cluster members
Solution Approach 1:
The patent segments the centralized HA protection state into individual per-VM protection states, with each VM's state persisted in its own namespace object within the distributed HC/OB storage system. This allows the HA functionality to adapt to distributed storage architecture while maintaining reliability through scattered storage of protection states across multiple hosts.
2Ease of operation
If conventional coarse-grained storage accessibility checks are used, then storage availability monitoring is simple, but the system cannot detect asymmetric accessibility in network partition scenarios where some hosts can access storage objects while others cannot
Solution Approach 1:
The patent divides the coarse-grained storage accessibility check into fine-grained per-VM accessibility checks. Instead of checking storage accessibility at the cluster level, the system checks accessibility for each individual VM's namespace object, enabling precise detection of asymmetric accessibility conditions in network partition scenarios.
Solution Approach 2:
The patent implements a feedback mechanism where the master HA module receives accessibility status information from slave HA modules about their ability to access VM storage objects. This feedback loop enables the master module to make informed decisions about VM failover based on real-time accessibility conditions across the cluster.
3Speed
If the system attempts VM restart without checking storage object accessibility, then failover response time is faster, but unnecessary restart attempts occur when storage objects are inaccessible, wasting resources and potentially causing system instability
Solution Approach 1:
The patent performs preliminary storage object accessibility checks before initiating VM restart operations. The master HA module determines whether VM storage objects are accessible to any host in the cluster before attempting failover, preventing wasted restart attempts and conserving system resources while maintaining fast response times for valid failover scenarios.
Data Source
AI summary
Techniques are disclosed for orchestrating high availability (HA) failover for virtual machines (VMs) running on host systems of a host cluster, where the host cluster aggregates locally-attached storage resources of the host systems to provide an object store, and where persistent data for one or more of the VMs is stored as per-VM storage objects across the locally-attached storage resources comprising the object store. In one embodiment, a host system in the host cluster executing a HA module determines a VM to be restarted on an active host system in the host cluster. The host system further determines if the VM's persistent data is stored in the object store. If so, the host system adds the VM to a list of VMs to be immediately restarted. Otherwise, the host system checks whether the VM is accessible to the host system by querying a storage layer of the host system configured to manage the object store.