BMC Node Fencing Coordination for Storage Split-Brain Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In high availability storage systems, synchronization flow failures between CPU nodes can lead to split-brain scenarios where each node corrupts shared data, and double fencing can cause complete storage system unavailability due to simultaneous fencing attempts.
Innovation Solution
A computer-implemented method where a first baseboard management controller (BMC) of a first node communicates with a second BMC of a second node to determine which node to fence from storage drives, using reset signals on storage drive ports or network controllers to isolate the fenced node, preventing double fencing and ensuring data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each CPU node attempts to fence the other node upon synchronization flow failure, then data corruption from split-brain scenario is prevented, but complete storage system unavailability occurs due to double fencing
Solution Approach 1:
The BMC establishes a preliminary communication channel before fencing is needed. When synchronization flow fails, the BMCs can communicate through this pre-established out-of-band channel to coordinate fencing, preventing the double-fencing race condition while maintaining system availability
Solution Approach 2:
The BMC acts as an intermediary between the CPU nodes and storage drives. It receives fencing requests from CPU nodes, communicates with other BMCs to determine which node should be fenced, and then executes the fencing action. This intermediary role prevents direct conflicting fencing actions between CPU nodes
2Speed
If CPU nodes directly fence each other upon detecting synchronization failure, then response time is reduced, but fencing race condition causes double fencing and system unavailability
Solution Approach 1:
The BMC serves as a mediator that receives fencing requests from CPU nodes and coordinates the actual fencing action. It communicates with other BMCs to determine which node should be fenced, preventing racing conditions while maintaining fast response through the out-of-band communication channel
Data Source
AI summary
A method, computer program product, and computing system for receiving, at a first baseboard management controller (BMC) of a first node, a fence request from a central processing unit (CPU) of the first node. The first BMC of the first node may communicate with a second BMC of a second node, wherein the first node and the second node are communicatively coupled to a plurality of storage drives. The first BMC and the second BMC may determine which node of the first node and the second node to fence from the plurality of storage drives. One of the first node and the second node may be fenced, via one of the first BMC and the second BMC, from the plurality of storage drives, thus defining a fenced node.