Opportunistic Failover in High Availability Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-availability cluster servers face delays in failover processes due to grace periods in lease-based locking mechanisms, which can lead to unacceptable access times and potential data inconsistency issues.
Innovation Solution
The opportunistic failover technique speeds up failover processing by concurrently assessing the status of the initial service instance and bringing up a failover instance while monitoring for access operations on resources, allowing quicker determination of false positives and freeing up resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the storage server waits for the grace period to pass before releasing the lock, then false positive failure indications are reduced, but failover delay increases
Solution Approach 1:
The failover instance is started in advance during the grace period before the lock is released. This preliminary action allows the service to be ready immediately upon lock acquisition, eliminating the startup delay that would otherwise occur after failover is initiated. The grace period is utilized productively to perform preparatory work rather than idle waiting.
Solution Approach 2:
Instead of having the system idle during the grace period, the failover instance continuously performs useful work by starting up and preparing to service requests. This continuous useful action transforms the previously wasted time into productive preparation, maintaining system readiness without compromising the lock holding period.
2Speed
If the failover instance starts up immediately upon lock acquisition, then service restoration is faster, but resource contention and data inconsistency risks increase
Solution Approach 1:
The failover instance is started in advance during the grace period while the initial instance still holds the lock. This ensures that when the lock is released and the failover instance needs to access resources, the service is already prepared and can immediately take over without additional startup delay, achieving fast service restoration.
Solution Approach 2:
The lock mechanism acts as an intermediary that coordinates access to resources. By waiting for the lock to be released before the failover instance accesses resources, the system ensures that only one instance accesses resources at a time, preventing data inconsistency while still achieving fast restoration through preliminary startup.
Data Source
AI summary
Failover processing of a service in a cluster of nodes is disclosed. A node brings up a failover instance of a service in response to an indication of a failure in an initial instance of the service. The failover service executes in the background so as not to be visible to users and thus does not receive user requests. The background service is terminated in response to the failure indication being a false positive. The background service is executed in the foreground in response to detecting that the initial instance of the service has failed and processes user requests.