Distributed Service Engine Failover via Autonomous Heartbeat Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed networking systems with active-active high availability (HA) configurations rely on a controller to reconfigure secondary nodes in case of primary node failure, but fail when the controller is unavailable, leading to system unavailability.
Innovation Solution
Secondary service engines autonomously detect primary service engine unavailability and select a new primary service engine from among themselves, even if the controller is unavailable, by monitoring heartbeats and using predefined priority lists or quorum-based selection methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a controller is used to monitor and reconfigure nodes in an active-active HA configuration, then the system can detect primary node failures, but the system becomes unavailable when the controller itself fails or cannot communicate with nodes
Solution Approach 1:
Secondary service engines autonomously detect primary node failures and perform self-reconfiguration without requiring controller intervention. Each secondary service engine monitors heartbeats from primary service engines and can independently assume the primary role when failures are detected, making the system self-healing and eliminating the single point of failure represented by the controller.
Solution Approach 2:
The failover detection and reconfiguration functionality is extracted from the centralized controller and distributed to individual secondary service engines. This allows secondary nodes to independently perform health monitoring and role assumption without relying on the controller, effectively removing the controller as a mandatory component for HA operation.
2Reliability
If secondary nodes autonomously detect failures and perform failover without a controller, then system availability is maintained during controller outages, but the mechanism for detecting failures and selecting new primaries becomes more complex
Solution Approach 1:
Service engines exchange periodic heartbeat messages to provide feedback on their operational status. Secondary service engines monitor these heartbeats from primary service engines, and the absence of heartbeats triggers autonomous failover. This feedback mechanism enables distributed failure detection without requiring complex centralized control logic.
Solution Approach 2:
Secondary service engines are pre-configured with priority lists indicating the order in which they should assume primary roles. When a failover is needed, the selected secondary service engine automatically consults its pre-configured priority list to determine which role to assume, eliminating the need for complex real-time negotiation or controller-mediated selection.
Data Source
AI summary
Providing high availability in a distributed networking platform includes detecting that an original primary service engine is unavailable, wherein: the original primary service engine and a plurality of secondary service engines are configured to provide one or more network applications associated with a virtual Internet Protocol (VIP) address; the original primary service engine and the plurality of secondary service engines are in active-active configuration mode; and the original primary service engine is configured to respond to Address Resolution Protocol (ARP) requests designating the VIP address. Additionally, providing high availability in a distributed networking platform further includes determining that a controller is unavailable; and configuring a selected secondary service engine as the next primary service engine.