Cluster Partner Takeover State for Disk Reservation Release
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network data storage systems face split-brain conflicts due to early release of disk reservations and delayed discovery of cluster information, leading to data corruption and system failures when storage servers attempt to access the same data.
Innovation Solution
Implementing a method that prevents a cluster partner from accessing and serving data when taken over, allowing early release of disk reservations and ensuring the rebooting node finds storage devices containing cluster information before proceeding with the boot process, using a new state in shared memory to manage takeover and giveback operations across the cluster interconnect.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If disk reservations are released early to reduce wait time, then client outage time is reduced, but split-brain conflicts may occur causing data corruption
Solution Approach 1:
The system performs preliminary actions by releasing disk reservations early while maintaining a takeover state that prevents the failed node from accessing storage devices. This allows the reservations to be released before the node fully reboots, reducing client outage time while preventing split-brain conflicts through the intermediary takeover state mechanism.
Solution Approach 2:
The takeover state acts as an intermediary mechanism between disk reservation release and full node recovery. It mediates the conflict by allowing reservations to be released while simultaneously preventing the failed node from accessing storage devices, thus enabling early reservation release without causing split-brain conflicts.
2Reliability
If the system waits for mailbox disks to be discovered before proceeding with boot, then takeover state information can be reliably read, but boot time is increased
Solution Approach 1:
The system performs preliminary actions by discovering and mounting mailbox disks early in the boot process, before the node attempts to access storage devices. This ensures takeover state information is reliably detected while minimizing the impact on overall boot time by handling disk discovery as a preliminary step.
Solution Approach 2:
The boot process is segmented into distinct phases: disk discovery phase, takeover state detection phase, and storage access phase. This segmentation allows the system to handle mailbox disk discovery separately and efficiently, ensuring reliable takeover state detection without significantly extending the critical path of the boot process.
3Loss of time
If storage devices are spun up quickly to reduce discovery time, then system recovery is faster, but storage device reliability may be compromised
Solution Approach 1:
The system performs preliminary actions by initiating storage device spin-up and discovery early in the boot process, before the node attempts to access or serve data. This allows storage devices to be prepared in advance with appropriate care for reliability, while still enabling fast recovery by having devices ready before they are needed.
Solution Approach 2:
The system dynamically adjusts the spin-up and discovery process based on the takeover state. When a node is taken over, storage devices are spun up and discovered with appropriate timing and care to ensure reliability, while the takeover state prevents premature data access that would require faster, potentially less reliable, device initialization.
Data Source
AI summary
An apparatus and a method that prevent a split-brain problem by preventing a cluster partner from accessing and serving data when the cluster partner is taken over by a storage server, while allowing early release of reservations on the cluster partner's storage devices before control is given back to the cluster partner.