Datacenter Activation Coordinator Preventing Split-Brain Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing clusters spanning physical locations face challenges in differentiating network failures from service failures, leading to potential split-brain scenarios and increased complexity and costs, especially in two-datacenter configurations, where managing site resilience and preventing downtime is crucial for maximizing service uptime.
Innovation Solution
The introduction of a Datacenter Activation Coordinator (DAC) that acts as a central point of control, preventing inappropriate activations by periodically querying servers and leveraging redundancy to detect datacenter failures, thereby preventing split-brain scenarios and ensuring timely recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a third datacenter is added as a witness to differentiate network failures from service failures, then the ability to determine active datacenter status is improved, but networking and facility costs increase
Solution Approach 1:
The patent extracts the essential function of the third witness datacenter and implements it as a virtual coordinator service running on existing datacenter infrastructure. This coordinator service provides the arbitration capability without requiring physical additional datacenter facilities, thereby reducing networking and facility costs while maintaining the ability to differentiate network failures from service failures.
Solution Approach 2:
The coordinator service is designed to perform multiple functions: it acts as a witness for failure detection, maintains the authorized list of active datacenters, coordinates failover decisions, and provides a centralized point of control. This multi-functionality replaces the need for dedicated witness infrastructure with a versatile service that operates on existing resources.
2Ease of operation
If manual activation procedures are implemented in the passive datacenter, then control over activation is improved, but operational complexity and recovery time increase
Solution Approach 1:
The passive datacenter is equipped with automated failover capabilities that allow it to self-activate when the active datacenter fails. The coordinator service automatically detects failures, updates the authorized list, and enables the passive datacenter to take over without manual intervention. This self-service mechanism reduces both operational complexity and recovery time while maintaining controlled activation through the coordinator's authorization system.
3Reliability
If the passive datacenter is activated during a power failure in the active datacenter, then service continuity is improved, but split-brain conditions occur when the active datacenter returns without coordination
Solution Approach 1:
The coordinator service implements a feedback mechanism where the passive datacenter must query the coordinator for authorization before activating services. The coordinator maintains an authorized list that explicitly permits or denies activation. This feedback loop ensures that even when the passive datacenter detects a failure and prepares to activate, it cannot actually activate services without coordinator authorization, preventing split-brain conditions while maintaining service continuity.
Solution Approach 2:
The coordinator service acts as an intermediary between the active and passive datacenters. It mediates the failover process by controlling which datacenter is authorized to provide services at any given time. This intermediary prevents direct uncoordinated activation by the passive datacenter, ensuring that only one datacenter can be active at a time, thus preventing split-brain scenarios while enabling seamless failover.
4Ease of operation
If operational procedures are manually injected into the two-datacenter solution, then control over failover is improved, but device complexity increases
Solution Approach 1:
The patent merges the failover control logic, authorization management, and state coordination into a single coordinator service. This consolidation simplifies the overall system architecture by replacing multiple manual procedures and distributed control mechanisms with a centralized service that handles all coordination tasks. The complexity is managed and hidden within the coordinator, presenting a simple interface to the datacenters.
Data Source
AI summary
A central controlling service for datacenter activation/deactivation control in a cluster deployment to assist in preventing a split-brain scenario. The central controlling service provides a central point of control in the datacenter for application servers to periodically query as to whether to go offline, online, or normal. Redundancy of the central service facilitates detection of datacenter failure by the redundant services interacting to resolve the state of control information. This control information is then used to answer the server queries. On startup from a datacenter failure, a single instance of the central service queries other redundant instance(s) to determine if the single instance is starting up from a datacenter-wide failure or from operations other than total datacenter failure. If the failure is datacenter-wide, a central service protocol assists in resolving to the single service keeping the associated datacenter servers offline; otherwise, the server queries are answered to go online.