Storage Node Failover Control During Maintenance Events
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Maintenance events in storage systems, particularly those not involving virtual machine reboots, can lead to reduced redundancy and availability, causing storage nodes to fail and disrupt data consistency, while reboots can result in insufficient operational storage nodes and emergency stops.
Innovation Solution
A storage system with failover capabilities among storage nodes, which includes an event monitoring and control mechanism to manage maintenance events by extending life/death monitoring timeouts, restricting I/O processing, and implementing freeze or blockage processes to maintain redundancy and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If maintenance events are performed on storage nodes, then system maintenance is achieved, but storage node failures occur due to life/death monitoring detecting stopped nodes as failures
Solution Approach 1:
The system performs preliminary actions by detecting maintenance events before they occur and preemptively adjusting life/death monitoring parameters. When a maintenance event is detected, the system changes the monitoring conditions in advance to prevent false failure detections, thereby maintaining storage node availability during maintenance operations.
Solution Approach 2:
The system changes the parameters of the life/death monitoring function based on maintenance event status. Specifically, it adjusts the monitoring timeout period or detection thresholds when maintenance is detected, allowing the storage node to remain operational during maintenance without being incorrectly identified as failed.
2Ease of manufacture
If storage nodes are stopped for maintenance not involving reboot, then maintenance is performed, but redundancy and availability are reduced
Solution Approach 1:
The system detects maintenance events in advance and preemptively adjusts monitoring parameters before the storage node stops. This preliminary action ensures that when the node stops for maintenance, it won't be incorrectly marked as failed, thereby maintaining redundancy and availability throughout the maintenance process.
Solution Approach 2:
The system implements feedback mechanisms where maintenance event information is continuously monitored and used to dynamically adjust life/death monitoring behavior. This feedback loop ensures that monitoring conditions adapt to current system state, preventing false failures and maintaining optimal redundancy during maintenance operations.
3Ease of manufacture
If storage nodes are stopped for reboot maintenance, then maintenance is performed, but insufficient operational nodes cause emergency stops
Solution Approach 1:
The system detects reboot maintenance events in advance and preemptively adjusts monitoring parameters before the storage node stops. This ensures that during reboot operations, the node won't be incorrectly marked as failed and other nodes can maintain sufficient operational capacity to prevent emergency stops.
Solution Approach 2:
The system changes monitoring parameters such as timeout periods or detection thresholds when reboot maintenance is detected. This parameter adjustment allows the system to tolerate the temporary unavailability of rebooting nodes while maintaining sufficient operational redundancy to prevent emergency stops.
4Measurement precision
If life/death monitoring is performed strictly, then failure detection accuracy is improved, but maintenance events are misdetected as failures
Solution Approach 1:
The system makes the life/death monitoring function dynamic by adjusting its parameters based on detected maintenance events. When maintenance is detected, the monitoring behavior changes to accommodate the temporary unavailability, preventing misdetection. When no maintenance is detected, strict monitoring ensures accurate failure detection, thus achieving both goals through dynamic adaptation.
Solution Approach 2:
The system changes monitoring parameters such as timeout periods or detection thresholds based on maintenance event status. This parameter adjustment allows the system to distinguish between planned maintenance (where nodes are intentionally stopped) and actual failures (where nodes should remain operational), thereby improving both detection accuracy and maintenance recognition.
Data Source
AI summary
The present invention has been made to reduce influence of maintenance events on performance. Disclosed is a storage system that includes a plurality of storage nodes, each having an arithmetic device and a memory. Upon detecting a failure of a separate storage node in the storage system, the plurality of storage nodes take over the failed storage node by failover. When a maintenance event occurs in the storage system, the plurality of storage nodes change, according to maintenance event information, conditions for detecting the failure of a storage node related to the maintenance event, and restrict data input/output processing. The maintenance event information is the information regarding the maintenance event.