Cluster Failure Chain Detection and Recovery via Ordered Membership
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing server cluster architectures face challenges in efficiently detecting and recovering from failures, leading to complex system consistency issues and increased maintenance complexity, especially when failures cascade and affect overall service availability.
Innovation Solution
A system and method for failure chain detection and recovery in a server cluster, utilizing a Group Management Service (GMS) client module that maintains an ordered view of group membership to determine recovery responsibilities, initiate recovery actions, and prevent failed nodes from rejoining until recovery is complete, with mechanisms for failure fencing and distributed state management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If servers are clustered to improve fault tolerance and scalability, then service availability and capacity are improved, but system complexity and maintenance difficulty increase
Solution Approach 1:
The patent introduces a load balancer as an intermediary component that manages server cluster operations. The load balancer handles failure detection, health monitoring, and request distribution, isolating the complexity from individual servers and providing centralized control over cluster behavior and failover mechanisms.
Solution Approach 2:
The load balancer serves multiple functions simultaneously: it distributes incoming requests across servers, monitors server health status, detects failures, and manages failover logic. This multi-functionality consolidates complex cluster management tasks into a single component, reducing overall system complexity while maintaining high availability.
2Reliability
If graceful failover is implemented with cluster-aware servers, then fault tolerance is improved, but design and maintenance complexity increase significantly
Solution Approach 1:
The patent extracts the complex failover management logic from individual servers and places it in the load balancer. Servers become simpler components that only need to respond to health checks and accept requests, while the load balancer handles the sophisticated failover detection, decision-making, and execution, thereby reducing design complexity on servers.
Solution Approach 2:
The load balancer acts as an intermediary that implements graceful failover mechanisms. It monitors server responses, detects failures through health checks, and manages the failover process by redirecting traffic to healthy servers, eliminating the need for complex cluster-aware logic in each server while maintaining fault tolerance.
3Loss of time
If failed servers are restarted quickly to improve service recovery, then downtime is reduced, but system consistency problems increase
Solution Approach 1:
The load balancer performs preliminary health checks and failure detection before initiating server restarts. It monitors server status continuously and can proactively identify failing servers, allowing for controlled restart timing that considers overall system state and maintains consistency by coordinating restarts with active failover processes.
Solution Approach 2:
The system implements feedback mechanisms where the load balancer continuously monitors server health and adjusts restart timing based on system state. It receives feedback about server performance and failure conditions, and uses this information to determine optimal restart moments that minimize consistency issues while reducing downtime through coordinated recovery actions.
Data Source
AI summary
A cluster or group of cooperating systems may implement failure chain detection and recovery. The group may include multiple nodes and each node may include a group management services (GMS) module that in turn may include a group communications mechanism to detect cluster membership events. Each GMS module may maintain an identically ordered view of the current group membership. When a member of the group fails, the member that joined the group immediately after the failed member, according to respective join times, may be selected to perform recovery operations for the failed member. If a group member fails while performing recovery operations for another failed member, the next member in the group (according to respective join times) may be selected to perform recovery for that node and may also perform recovery operations for the original failed node as well.