Dynamic Timeout Adjustment for Storage Node Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face challenges in determining an optimal time-out value for node failure detection during failover, as it is influenced by various system environments, leading to prolonged IO stop times or erroneous failure detection.
Innovation Solution
A storage system that monitors network status between nodes to dynamically adjust time-out values based on current network conditions, ensuring efficient failover while preventing erroneous failure detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the time-out time for node failure detection is set to a fixed value, then the system is simple to operate, but the failover time is prolonged when network load is heavy or the time-out time is too long
Solution Approach 1:
The patent applies dynamics by transitioning from a fixed time-out value to a dynamic time-out value that automatically adjusts based on network status. The system monitors network conditions (such as packet loss, latency, or congestion) and modifies the time-out value accordingly, allowing it to extend during heavy network load to prevent false failures while maintaining quick failover during normal conditions.
Solution Approach 2:
The patent changes the parameter of time-out value from a static configuration to a dynamic parameter that varies with network conditions. By monitoring network status and adjusting the time-out value as a variable parameter, the system optimizes failover performance without requiring manual intervention or complex configuration.
2Loss of time
If the time-out time for node failure detection is set to a short value, then the failover time is reduced, but erroneous detection of node failure occurs when network load is heavy
Solution Approach 1:
The system dynamically adjusts the time-out value based on real-time network status monitoring. When network load is detected to be heavy (through metrics like packet loss or latency), the time-out value is extended to prevent erroneous failure detection. When network conditions are normal, the time-out value is reduced to enable quick failover, thus maintaining both speed and accuracy.
Solution Approach 2:
The patent implements feedback by continuously monitoring network status and using this information to adjust the time-out value. The system receives feedback from network condition measurements and modifies its behavior accordingly, preventing false positives during congestion while maintaining sensitivity during normal operation.
3Reliability
If the time-out time for node failure detection is set to a long value, then erroneous failure detection is avoided, but the IO stop time from host is increased
Solution Approach 1:
The system uses dynamic adjustment of the time-out value based on network conditions to balance reliability and speed. During normal network operation, a shorter time-out value minimizes IO stop time. When network congestion is detected, the time-out value is extended to avoid false failures, thus preventing unnecessary IO interruptions while maintaining detection accuracy.
Solution Approach 2:
The patent changes the time-out parameter from a fixed long value to a dynamic parameter that adapts to network conditions. This allows the system to use shorter time-out values during normal conditions to minimize IO stop time, while extending the value only when necessary to prevent false failures, optimizing the trade-off between speed and reliability.
Data Source
AI summary
In a storage system, a failover time is shortened while avoiding erroneous detection of a node failure according to network quality. The storage system includes a plurality of storage nodes having processors; a data storage device; the plurality of storage nodes having the processors for processing data to be input to and output from the storage device; and a network for connecting the plurality of storage nodes, wherein the plurality of storage nodes monitors the operation status of each other to perform node failure detection for determining the occurrence of a failure of the storage node on the basis of a time-out value, performs a failover process in which one of the other storage nodes takes over the process of the storage node when the failure occurs in the storage node, and adjusts the time-out value on the basis of the status of the network between the storage nodes.