Multi-AZ Cluster Remediation for Degraded Availability Zones
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions fail to efficiently detect degraded availability zones in multi-AZ SDDCs and remediate affected software components, leading to inefficient migration of software components across availability zones.
Innovation Solution
A system and method that automatically detects a degraded state of an availability zone based on host failure events, scales out a second cluster section in a non-degraded zone, and upon recovery, scales in the second cluster section back to the original zone, using an elastic resource scheduler and analytic service to manage multi-AZ clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual detection and remediation methods are used for degraded AZs, then system complexity is reduced, but detection efficiency and remediation speed deteriorate
Solution Approach 1:
The system implements self-service through automated detection and remediation mechanisms. The elastic resource scheduler automatically detects degraded AZs by monitoring host failure events and triggers remediation actions without human intervention. The system self-manages the entire lifecycle from detection to software component migration and back, eliminating the need for manual operational intervention while maintaining high detection efficiency.
Solution Approach 2:
The system employs feedback mechanisms where the elastic resource scheduler continuously monitors host failure events and AZ health status. When degradation is detected, the system initiates remediation and receives feedback through scale-in operations that confirm recovery. This closed-loop feedback system enables automated detection and remediation while managing complexity through structured event-driven architecture.
2Reliability
If automated detection and remediation systems are implemented, then detection efficiency and remediation speed improve, but system complexity increases
Solution Approach 1:
The system segments the complex remediation process into distinct modular components: the elastic resource scheduler for detection, the software component migration mechanism for remediation, and the scale-in operation for verification. Each component handles a specific aspect of the AZ degradation response, making the overall complex system manageable through clear separation of concerns while maintaining high availability through coordinated operation of these segments.
Solution Approach 2:
The elastic resource scheduler acts as an intermediary that bridges detection and remediation functions. It receives host failure events, determines AZ degradation status, triggers software component migration to healthy AZs, and coordinates with scale-in operations to verify recovery. This intermediary layer manages system complexity by centralizing the decision-making logic while enabling automated reliability maintenance.
3Reliability
If software components are migrated during AZ degradation, then fault tolerance is maintained, but migration overhead and performance impact increase
Solution Approach 1:
The system performs preliminary actions by maintaining software component replicas across multiple AZs before degradation occurs. When an AZ degrades, the pre-positioned replicas in healthy AZs can immediately take over without requiring time-consuming migration operations. This preliminary replication strategy ensures fault tolerance while minimizing migration time and performance impact during actual degradation events.
Data Source
AI summary
System and computer-implemented method for managing multi-availability zone (AZ) clusters of host computers in a cloud computing environment automatically detects a degraded state of a first AZ in the cloud computing environment based on host failure events for host computers in a first cluster section of a multi-AZ cluster of host computers located in the first AZ and a recovered state of the first AZ based a successful scale-in operation of another multi-AZ cluster located partially in the first AZ. In response to the detection of the degraded state of the first AZ, a second cluster section of the multi-AZ cluster of host computers located in a second AZ is scaled out. In response to the detection of the recovered state of the first AZ, the second cluster section of the multi-AZ cluster of host computers located in the second AZ is scaled in.