Health-Status Server Prevents Split-Brain in High-Availability Clusters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

High-availability clusters face split-brain scenarios due to partitioning events that isolate nodes, leading to data corruption and service unavailability, as standby nodes mistakenly assume active nodes have failed.

Innovation Solution

Implementing a system that detects partitioning events and broadcasts cluster-health messages from a health-status server to nodes, allowing them to react based on the health status of other nodes, preventing mistaken failovers and ensuring continuous service.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If nodes use heartbeat messages to detect active-node failure, then failover detection capability is improved, but false failure detection occurs during partitioning events causing split-brain scenarios

Engineering Contradiction:

Improvefailure detection accuracyVSAvoidservice availability

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent introduces a cluster coordinator as an intermediary component that mediates between nodes during partitioning events. The coordinator receives heartbeat messages from all nodes, determines the actual cluster state, and provides authoritative failure declarations to prevent false failovers. This intermediary resolves the split-brain scenario by serving as the ultimate authority on node health status during network partitions.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent implements enhanced feedback mechanisms where nodes receive explicit failure declaration messages from the cluster coordinator before initiating failover. Instead of reacting solely to missing heartbeats, nodes continuously receive feedback from the coordinator about the actual cluster state, ensuring failover decisions are based on accurate information rather than transient network conditions.

Inventive Principle:

Principle #23Feedback

2Productivity

If standby nodes immediately assume computing tasks upon detecting active-node failure, then service continuity is improved, but data corruption occurs due to premature failover during partitioning

Engineering Contradiction:

Improveservice continuityVSAvoiddata corruption

Core Design Contradiction:

ProductivityVSObject-affected harmful factors

Solution Approach 1:

The patent implements preliminary verification actions before failover execution. When a standby node detects potential active-node failure, it first queries the cluster coordinator for authoritative failure declaration before assuming computing tasks. This preliminary check prevents premature failover during partitioning events while still enabling rapid failover when actual failures occur, thus avoiding data corruption.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent applies preliminary anti-action by having standby nodes refrain from failover until the cluster coordinator explicitly declares the active node failed. This counter-action to the natural failover impulse prevents harmful premature failovers during partitioning events, while the coordinator's failure declaration mechanism ensures that legitimate failures are still addressed promptly.

Inventive Principle:

Principle #9Preliminary anti-action

3Adaptability or versatility

If nodes operate independently during network partitioning, then autonomous operation is improved, but split-brain scenarios occur due to lack of coordination

Engineering Contradiction:

Improveautonomous operation capabilityVSAvoidcluster state consistency

Core Design Contradiction:

Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The patent maintains cluster state consistency during partitioning by having the cluster coordinator serve as the ultimate mediator that all nodes reference for failure decisions. Even when nodes operate autonomously during network partitions, they independently query the coordinator for authoritative state information, ensuring that split-brain scenarios cannot occur as long as the coordinator remains accessible to at least one partition.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent inverts the traditional approach by having nodes seek authorization from the cluster coordinator before initiating failover, rather than autonomously deciding based on local observations. This inversion ensures that cluster state consistency is maintained because the coordinator, which has visibility into the entire cluster, makes the final determination on which nodes should failover.

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS10114713B1Systems and methods for preventing split-brain scenarios in high-availability clusters

Publication Date: 2018.10.30 JUNIPER NETWORKS INC

AI summary

A computer-implemented method for preventing split-brain scenarios in high-availability clusters may include (1) detecting, at a first node of a high-availability cluster, a partitioning event that isolates the first node from a second node of the high-availability cluster, (2) broadcasting, from a health-status server and after the partitioning event has occurred, a cluster-health message to the first node that includes at least a health status of the second node that is based on whether the health-status server received a node-health message from the second node, and (3) reacting, at the first node and based at least in part on whether the first node received the cluster-health message, to the partitioning event such that the partitioning event does not result in a split-brain scenario within the high-availability cluster. Various other methods, systems, and computer-readable media are also disclosed.