Master Node Designation in Non-HA Cluster Services
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natively non-high availability services, such as database services, face challenges in maintaining resilience and accuracy during deployment crashes, often requiring manual intervention due to incorrect master/slave node designation, leading to potential data loss or failure to establish a master node.
Innovation Solution
A system that automatically improves resilience by determining lifecycle events, storing master node role and timestamp information, and comparing timestamps across nodes to accurately designate the master node, leveraging existing infrastructure and communication networks for efficient recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual intervention is used to establish master/slave node designation during deployment crashes, then accuracy of master node selection can be maintained, but productivity and ease of operation deteriorate due to requiring manual intervention
Solution Approach 1:
The system performs preliminary actions by storing timestamp information and role designations in persistent storage before deployment crashes occur. When a crash happens, this pre-stored information is automatically retrieved and used to determine master node designation, eliminating the need for manual intervention while maintaining accuracy.
Solution Approach 2:
The system enables self-service by implementing automated scripts that independently determine master node designation using timestamp comparison. The nodes automatically recover without requiring external manual intervention, as the system uses its own stored timestamp data to make recovery decisions.
2Productivity
If automated failover scripts are implemented, then productivity improves by eliminating manual intervention, but reliability deteriorates due to potential incorrect master node designation after deployment crashes
Solution Approach 1:
The system performs preliminary actions by storing timestamp information and role designations in persistent storage before deployment crashes occur. When a crash happens, this pre-stored information is automatically retrieved and used to determine master node designation, eliminating the need for manual intervention while maintaining accuracy.
Solution Approach 2:
The system implements feedback by having nodes compare their timestamp information with other nodes to verify master node designation. This feedback mechanism ensures that the automated script selects the correct master node based on the most recent timestamp, improving reliability of the automated process.
3Reliability
If timestamp information is stored in persistent storage, then reliability improves by enabling accurate master node designation after crashes, but device complexity increases due to additional storage and comparison mechanisms
Solution Approach 1:
The system applies segmentation by separating the failover mechanism into distinct components: timestamp generation, persistent storage, timestamp retrieval, and timestamp comparison. This modular approach reduces overall system complexity by making each component simple and independent while maintaining high reliability through their coordinated operation.
Solution Approach 2:
The system performs preliminary actions by storing timestamp information and role designations in persistent storage before deployment crashes occur. When a crash happens, this pre-stored information is automatically retrieved and used to determine master node designation, eliminating the need for manual intervention while maintaining accuracy.
Data Source
AI summary
Some embodiments provide a system and method associated with improving resilience for a non-high availability service provided by a multi-node cluster (e.g., a bosh-based database service). A processor of a node in the cluster may determine that a lifecycle event has occurred (e.g., an initial deployment, failover, or deployment update). Responsive to the lifecycle event, the processor may store information about a role of a master node and timestamp information (e.g., on persistent storage). The processor may subsequently determine that a cluster deployment crash has occurred for the service. Timestamp information associated with multiple nodes may then be compared to establish one of the nodes as the master node. For example, the most recent timestamp may be used to establish the master node.