Coordination Point Reachability Matrix for I/O Fencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In high-availability distributed storage systems, the unavailability of coordination points during fencing operations can lead to data integrity issues and data corruption, as well as application failover, due to the potential for split-brain conditions and sub-optimal fencing race results, especially in asymmetric/FSS systems where not all nodes have access to coordination points.
Innovation Solution
The implementation of a coordination point reachability matrix (CPRM) that is generated and transmitted among nodes to determine the accessibility of coordination points, allowing nodes to join or leave the cluster while ensuring access to a majority of coordination points, thereby preventing split-brain conditions and optimizing fencing operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If coordination points are used for fencing operations, then data integrity is protected, but system availability deteriorates when coordination points are unavailable
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing reachability information in CPRMs before fencing operations are needed. Each node determines in advance which coordination points it can reach and stores this information. When a fencing operation is required, the pre-computed CPRMs allow rapid determination of the winning partition without requiring real-time access to all coordination points, thus maintaining both data integrity and system availability.
Solution Approach 2:
The CPRM acts as an intermediary data structure that mediates between the physical coordination points and the fencing decision-making process. Instead of directly querying coordination points during fencing operations, nodes use the CPRM to determine reachability and make fencing decisions. This intermediary layer allows the system to maintain data integrity through coordination point validation while avoiding availability issues by using pre-computed information.
2Measurement precision
If all nodes must access all coordination points, then fencing accuracy is improved, but device complexity increases in asymmetric systems
Solution Approach 1:
The system applies local quality by allowing different nodes to have different reachability characteristics to coordination points. Instead of requiring uniform access patterns, each node's CPRM reflects its specific reachability to coordination points. The fencing decision is made based on the aggregate reachability of the partition, not on individual node characteristics, thus maintaining fencing accuracy while accommodating asymmetric system configurations.
Solution Approach 2:
The system changes the parameter of reachability from a binary yes/no for each node-coordination point pair to an aggregate measure at the partition level. The CPRM allows the system to evaluate fencing decisions based on whether a partition can reach a majority of coordination points collectively, rather than requiring each individual node to reach all coordination points. This parameter transformation simplifies asymmetric system configurations while maintaining fencing accuracy.
3Reliability
If fencing operations wait for coordination point availability, then data integrity is ensured, but response time increases
Solution Approach 1:
The system performs preliminary actions by pre-determining and storing reachability information in CPRMs during normal operation or system initialization. When a fencing operation is triggered, the system does not need to wait for coordination points to become available or to perform new reachability tests. Instead, it uses the pre-computed CPRMs to immediately determine which partition can reach the majority of coordination points, thus ensuring data integrity while minimizing fencing response time.
4Device complexity
If coordination points are centralized, then fencing control is simplified, but single points of failure increase
Solution Approach 1:
The system applies segmentation by dividing the coordination point infrastructure into multiple distributed coordination points rather than using a single centralized authority. The CPRM tracks reachability to multiple coordination points individually. During fencing operations, the partition that can reach the majority of distributed coordination points wins, which simplifies control (any partition can independently determine victory) while eliminating single points of failure (the system tolerates failures of individual coordination points).
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Various systems, methods, and processes to optimize input/output (I/O) fencing operations in systems that implement coordination points are presented. A matrix is generated. The matrix includes information indicating the accessibility of one or more coordination points by a node. The method then transmits the matrix to one or more other nodes. In addition, in response to determining that a node is no longer part of a sub-cluster, another node in the sub- cluster is designated as a racer node. The racer node accesses a matrix, and the matrix includes information to determine whether a majority of coordination points in a cluster are accessible by nodes in the sub-cluster. Based on the accessing, a determination is made that the information indicates that the majority of coordination points are accessible by the nodes. The information is then broadcasted. Further, in response to a cluster being partitioned into multiple network partitions, a determination is made, using a first node in a first network partition, whether the first node wins one or more coordination points. The determination of the first node is then transmitted to a second node in the first network partition. Further, another determination is made using the second node whether the second node wins one or more other coordination points other than the one or more coordination points won by the first node.