Persistent Flag for Storage Node Switchover Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High availability data storage systems face challenges in seamless failover and switchback operations during node failures, particularly in disaster recovery scenarios, where data integrity and minimal disruption are critical, and existing solutions lack efficient mechanisms to track progress and handle failures during these operations.
Innovation Solution
The implementation of a flag mechanism that persists across reboots to track the progress of switchover or switchback operations, allowing nodes to determine the status and recover quickly from failures by identifying the cause of the failure, such as power loss or panic, and adjusting the recovery process accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a node fails during switchover or switchback operation, then the system experiences disruption and potential data loss, but without a flag mechanism the node cannot quickly identify its status and recover
Solution Approach 1:
The flag is set in advance during switchover or switchback operations to record the node's status and progress before a failure occurs. This preliminary action ensures that when failure happens, the information is already in place for rapid recovery without needing to re-determine the status from scratch.
Solution Approach 2:
The flag creates a copy of the node's status information that persists across reboots. This copy allows the node to quickly identify its previous state and recovery progress without accessing complex operational logs or metadata, significantly accelerating the recovery process.
2Ease of operation
If the node reboots after failure, then the system can recover, but without flag persistence the node must re-determine its status from scratch increasing recovery time
Solution Approach 1:
The flag records the node's status and progress before failure, so when the node reboots this information is immediately available. The node doesn't need to re-determine its status from scratch or search through operational logs, making recovery simpler and faster.
Solution Approach 2:
The flag mechanism enables the node to self-identify its status and recovery progress without external intervention or complex status determination procedures. The node uses the flag information to automatically determine what actions to take during recovery, simplifying the operational process.
3Reliability
If the flag is not persisted across reboots, then the system uses less storage, but the node cannot maintain status information needed for rapid recovery
Solution Approach 1:
The flag extracts and isolates only the essential status information needed for recovery from the complex node state. By separating this critical information into a dedicated persistent flag, the system maintains reliable status tracking without needing to store the entire node state or complex operational metadata.
Solution Approach 2:
The flag provides localized persistence for status information, allowing different parts of the system to have different persistence characteristics. Only the critical recovery status is persisted in the flag, while other node state information can be handled differently, optimizing storage usage.
4Difficulty of detecting and measuring
If the node cannot identify its status quickly after failure, then the system lacks visibility into recovery progress, but with flag tracking the node can rapidly determine its status and adjust recovery accordingly
Solution Approach 1:
The flag creates a persistent copy of the node's status information that is immediately available after failure. This copy eliminates the need to re-determine status from operational logs or metadata, significantly reducing status determination time and making the status easily detectable.
Solution Approach 2:
The flag provides immediate feedback to the node about its previous status and progress. This feedback mechanism allows the node to quickly understand where it left off and adjust its recovery actions accordingly, eliminating delays in status detection and determination.
Data Source
AI summary
Techniques for recovering from a failure at a disaster recovery site are disclosed. An example method includes receiving an indication to shift control of a set of volumes of a plurality of volumes. The set of volumes is originally owned by a second storage node. The first storage node is a disaster recovery partner of the second storage node. The method includes shifting control of the set of volumes. The method further includes during the shifting, changing a status of a flag corresponding to a progress of the shifting. The method also includes during a reboot of the first storage node, determining the status of the flag and determining, based on the status of the flag, whether to mount the set of volumes during reboot at the first storage node.