Quorum Service for Data Storage Cluster Split Brain Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In clustered data storage systems, the existing TTL-based polarization algorithm fails to handle preferred array failures, leading to potential data corruption due to 'split brain' scenarios where both arrays keep their sides of the stretched volume online without replicating changes, which can result in data corruption.
Innovation Solution
The introduction of a quorum service that allows the non-preferred array to establish quorum before taking the stretched volume offline, ensuring that only one array continues to service host I/O requests and preventing data corruption by selecting a survivor node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If both arrays continue to service host I/O requests without quorum service, then availability is maintained, but data corruption occurs due to split brain scenarios
Solution Approach 1:
A quorum service is introduced as an intermediary component between the two storage arrays. The quorum service receives registration requests from both arrays and processes auto-promote requests to determine which array should service host I/O requests. This mediator prevents split brain scenarios by ensuring that only one array can be in the auto-promoted state at any given time, thereby maintaining data integrity without significantly increasing system complexity.
2Reliability
If quorum service is introduced to prevent split brain, then data integrity is protected, but system complexity increases
Solution Approach 1:
The quorum service operates autonomously by automatically processing registration requests from both arrays and handling auto-promote requests based on predefined logic. The service self-manages the quorum state, determining which array should service host I/O requests without requiring manual intervention. This self-service approach protects data integrity while minimizing the operational complexity burden on system administrators.
3Ease of operation
If TTL-based polarization algorithm is used, then preferred array failure is not handled properly, but simpler operation is maintained
Solution Approach 1:
The quorum service performs preliminary actions by pre-registering both arrays before any failure occurs. When a failure happens, the quorum service is already in position to immediately process auto-promote requests and determine the surviving array. This preliminary setup ensures that preferred array failures are handled properly without complicating the operational simplicity, as the quorum service automatically manages the failure response.
Data Source
AI summary
In a protective quorum service, during an initial period of normal operation in which a clustered pair of data storage nodes provide host I/O access to a data storage object and replicate write-type requests to each other, the nodes are first registered to the quorum service. Subsequently, based on the registration and in response to a first auto promote request from a first-requesting node, a success response is returned and the service enters an auto promoted condition, the success response indicating that the first-requesting node is to continue providing the host I/O access to the data storage object without write replication. In response to receiving a subsequent auto promote request from the other node when in the auto promoted condition, a failure response is returned indicating that the other node is to cease providing host I/O access to the data storage object.