Automated Master Failover via Service Coordinator
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Switching mastership between single-master services in multiple data centers is a time-consuming process that affects availability and can lead to degraded user experiences due to the need for manual configuration changes and service restarts, especially when multiple services are involved, potentially causing data integrity issues and service outages.
Innovation Solution
Implementing a service coordinator that notifies instances of the current master service of the status change, ensuring all instances acknowledge the change before designating the new master, thereby avoiding simultaneous active states of the old and new master services without requiring service shutdowns or restarts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual configuration changes and service restarts are performed to switch mastership, then data integrity is maintained, but service availability is degraded and user experience is impacted
Solution Approach 1:
The patent replaces the manual mechanical process of configuration changes and service restarts with an automated notification-based system. The service coordinator automatically notifies slave instances of master status changes, eliminating the need for manual intervention and service disruption while maintaining data integrity through coordinated state management.
Solution Approach 2:
The slave instances autonomously manage their state transitions based on notifications from the service coordinator. When notified of a master status change, slave instances automatically adjust their operational state without requiring manual configuration changes or service restarts, thereby maintaining availability while ensuring integrity.
2Ease of operation
If manual configuration changes are made to switch mastership, then service control is maintained, but the process takes significant time (e.g., one hour) and requires service shutdowns
Solution Approach 1:
The system performs preliminary actions by having slave instances continuously monitor and wait for notifications from the service coordinator about master status changes. This preparation allows instant state transitions without time-consuming manual configuration changes, reducing mastership switching time from hours to minutes while maintaining operational control.
Solution Approach 2:
The manual mechanical process of configuration changes and service restarts is replaced with an automated notification-based system. The service coordinator automatically notifies slave instances of master status changes, eliminating the need for manual intervention and service disruption while maintaining data integrity through coordinated state management.
3Reliability
If multiple services are involved in mastership switching, then comprehensive service coordination is achieved, but the complexity and time required increase significantly
Solution Approach 1:
The service coordinator performs multiple functions including tracking master status, notifying slave instances, and coordinating state transitions across all services. This universal coordination mechanism handles multiple services through a single unified process, reducing the complexity of managing individual service transitions while ensuring comprehensive coordination and reliability.
Data Source
AI summary
Techniques for switching mastership from one service in a first data center to a second (redundant) service in a second data center are provided. A service coordinator in the first data center is notified about the master switch. The service coordinator notifies each instance of the first service that the first service is not a master. Each instance responds with an acknowledgement. After it is confirmed that all instances of the first service have responded with an acknowledgement, a client coordinator in the first and/or second data center is updated to indicate that the second service is the master so that clients may send requests to the second service. Also, a service coordinator in the second data center is notified that the second service is the master. The service coordinator notifies each instance of the second service that the second service is the master. Each instance responds with an acknowledgement.