Coordinator Service for Cluster Membership Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software-defined data centers face challenges in managing cluster membership in a simple, reliable, and automated manner, especially when nodes are added or removed, due to the limitations of coordination services like Zookeeper, which require manual recovery and inconsistent membership views.
Innovation Solution
A coordinator service that automatically negotiates cluster membership changes at runtime, maintaining consistent views by persisting node lists and member lists in nonvolatile memory, and using a unique change ID for reconfiguration requests, ensuring all nodes acknowledge changes before updating the member list and notifying them of updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If coordination services like Zookeeper are used to handle cluster reconfiguration, then cluster membership management can be handled, but manual recovery is required when the coordination service loses its quorum and automation is lost
Solution Approach 1:
The system implements self-service through the stable node that automatically performs recovery operations. When the coordinator becomes unavailable, a stable node with a valid member list automatically assumes the coordinator role and sends reconfiguration requests to nodes, eliminating the need for manual intervention to restore cluster functionality.
Solution Approach 2:
The system performs preliminary action by persisting the member list in non-volatile storage on stable nodes before failures occur. This pre-stored information enables automatic recovery without manual intervention, as the stable node can reconstruct the coordinator's state using the persisted member list from previous operational states.
2Reliability
If coordination services are used for cluster reconfiguration, then membership changes can be managed, but the system complexity increases due to additional coordination layers
Solution Approach 1:
The system extracts the critical coordination function from a complex external service like Zookeeper and implements a simplified internal coordination mechanism. The coordinator and stable nodes work together with basic message passing and member list persistence, removing the need for complex coordination service infrastructure while maintaining membership consistency.
Solution Approach 2:
The stable node acts as an intermediary between the coordinator and cluster nodes during recovery scenarios. It receives the member list from the coordinator, stores it persistently, and uses it to mediate reconfiguration requests, simplifying the coordination architecture by introducing a single intermediary rather than relying on complex coordination services.
3Reliability
If manual management of cluster membership is performed, then control is maintained, but operational efficiency decreases and time is lost
Solution Approach 1:
The system implements self-service by automatically detecting coordinator failures and performing recovery through stable nodes. The automatic detection and recovery process eliminates manual intervention requirements, maintaining reliable membership control while significantly improving operational efficiency by removing time-consuming manual steps.
Solution Approach 2:
The system uses feedback mechanisms where nodes monitor coordinator availability and automatically trigger recovery procedures when the coordinator becomes unavailable. This feedback-driven automatic response maintains membership control reliability while improving productivity by eliminating manual monitoring and intervention requirements.
Data Source
AI summary
A method is provided for a coordinator to manage cluster membership. In a stable state, the coordinator provides a member list to all the nodes in a node list. The member list includes nodes that are cluster members. The node list includes nodes that are or wish to be members of the cluster. When the node list differs from the member list, the coordinator advances to a reconfiguration state to change the membership of the cluster. In the reconfiguration state, the coordinator sends a reconfiguration request to all the nodes in the node list. When reconfiguration acknowledgements are received from all the nodes within a timeout period, the coordinator updates the member list to be equal to the node list, persists the updated member list, sends a reconfiguration confirmation including the updated member list to all the nodes in the node list, and returns to the stable state.