Storage System Heartbeat Monitoring for Failover Resumption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multi-site distributed data storage systems face challenges in reducing resumption time for I/O operations during automatic unplanned failover from a primary to a secondary mirrored storage system, due to delays in failure detection, serial health-check operations, redundant operations, and system resource contention.
Innovation Solution
The solution involves monitoring heartbeat information at ultra-short intervals, optimizing health check operations through multi-level parallelization, avoiding redundant operations using caching, reducing contention by batching operations, minimizing system calls and context switches through prefetching, and facilitating early-abort of conflicting workflows.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If heartbeat information is monitored at traditional time intervals during failover, then system resource consumption is reduced, but failure detection time increases leading to longer I/O resumption time
Solution Approach 1:
The system dynamically adjusts heartbeat monitoring intervals based on system state. During normal operation, traditional intervals are used to conserve resources. During failover detection, the interval automatically shortens to ultra-short intervals for rapid detection, then returns to normal after failover completion, optimizing the balance between detection speed and resource consumption throughout the operational cycle
Solution Approach 2:
The system prepares ultra-short interval monitoring in advance by having the capability ready but not actively executing it during normal operation. When failover conditions are detected, the pre-configured ultra-short interval mechanism is immediately activated, enabling rapid failure detection without the overhead of continuously running high-frequency monitoring
2Loss of time
If health check operations are performed serially during failover, then system complexity is reduced, but failover latency increases
Solution Approach 1:
The health check operation is divided into multiple independent parallel tasks that can execute simultaneously. Different health check functions (e.g., connectivity checks, data integrity checks, service status checks) are segmented into separate operations that run in parallel, reducing total failover latency while maintaining manageable complexity through modular task design
Solution Approach 2:
The system transitions from single-threaded sequential health checking to multi-threaded parallel health checking, adding the dimension of concurrent execution. This dimensional change allows multiple health check operations to proceed simultaneously rather than one after another, significantly reducing failover latency while the threading infrastructure manages the increased complexity
3Reliability
If redundant health check operations are performed during failover, then failure detection accuracy is improved, but failover latency increases
Solution Approach 1:
Health check results and system state information are cached in advance during normal operation. When failover is triggered, the cached information is immediately utilized to avoid redundant checks, providing accurate failure detection without the time penalty of repeating all health check operations, thus reducing failover latency while maintaining detection accuracy
Solution Approach 2:
The system discards redundant health check operations by utilizing previously obtained health check results and cached system state information. Instead of performing all health checks again during failover, the system recovers and reuses valid cached data, eliminating unnecessary repetition while maintaining accurate failure detection
Solution Approach 3:
During failover, the system performs a selective subset of health check operations rather than all possible checks. Critical path health checks are executed immediately while less critical checks are deferred or skipped, providing sufficient detection accuracy for the failover context without the full time cost of comprehensive checking
4Loss of time
If system calls and context switches are频繁 performed during failover, then operational flexibility is improved, but I/O resumption time increases
Solution Approach 1:
Critical data and configuration information are pre-loaded into memory and prepared in advance before failover occurs. During the actual failover process, the system operates with this pre-loaded information, minimizing the need for additional system calls and context switches. This preliminary preparation reduces I/O resumption time while the pre-loaded data maintains operational flexibility
Solution Approach 2:
Multiple system calls and context switch operations are merged and batched together during failover. Instead of performing numerous individual system calls sequentially, the system combines related operations into fewer bulk operations, reducing the total number of context switches and thereby reducing I/O resumption time while maintaining the necessary operational flexibility through the combined operations
Data Source
AI summary
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during an automatic unplanned failover (AUFO). A computer-implemented method includes monitoring, with a second cluster, heartbeat information received at ultra-short time intervals from a first connection of one or more storage objects of the first cluster, determining, with the second cluster, whether the heartbeat information from the first connection is received during an ultra-short time interval, and intelligently routing heartbeat information from the one or more storage objects of the first cluster from the first connection to a second connection when the heartbeat information from the first connection is not received during the ultra-short time interval.