Storage Cluster Failure Detection via Remote Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage cluster failure detection methods are inadequate in ensuring rapid and reliable data availability, particularly in distributed storage systems, as they often fail to detect failures within the specified recovery point objective (RPO) and recovery time objective (RTO) requirements, leading to potential data loss and system downtime.
Innovation Solution
A metrocluster configuration with high availability pairs and a backup cluster is implemented, where each node in a primary cluster is mirrored in a backup cluster, enabling swift data replication and failover in case of primary cluster failure, utilizing a remote failure detect module for monitoring and verification of node and cluster connectivity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous mirroring is implemented between primary and backup clusters, then data availability and reliability are improved, but device complexity increases
Solution Approach 1:
The system segments the storage infrastructure into primary clusters and backup clusters, each independently manageable. This segmentation allows the complexity of synchronous mirroring to be distributed across multiple independent units rather than concentrated in a single system, making the overall complex configuration more manageable through modular deployment and independent operation of each cluster segment.
Solution Approach 2:
A remote failure detect module serves as an intermediary that monitors cluster health and triggers failover operations. This intermediary component simplifies the complexity of direct cluster-to-cluster monitoring by providing a centralized, standardized interface for detecting failures and initiating recovery procedures, thereby reducing the operational complexity of maintaining synchronous mirroring.
2Loss of time
If rapid failure detection is implemented to meet RTO requirements, then recovery time is reduced, but measurement precision requirements increase
Solution Approach 1:
The system implements continuous feedback monitoring through the remote failure detect module, which constantly checks cluster accessibility and node status. This feedback mechanism enables rapid detection of failures by immediately detecting when a primary cluster becomes inaccessible, triggering the failover process. The feedback loop is tuned to detect failures within the RTO window while filtering out false positives through configurable thresholds and verification mechanisms.
Solution Approach 2:
The system performs preliminary actions by pre-configuring backup clusters with mirrored data and pre-establishing failover protocols before failures occur. This preliminary preparation ensures that when a failure is detected, the system can immediately execute the pre-planned recovery sequence, reducing the time required for detection and response while maintaining precise failure identification through pre-defined monitoring criteria.
3Reliability
If multiple monitoring mechanisms are deployed to verify cluster status, then detection reliability is improved, but device complexity increases
Solution Approach 1:
The system merges multiple monitoring functions into a single remote failure detect module that can monitor multiple clusters and nodes through standardized protocols. This consolidation combines the reliability benefits of multiple monitoring mechanisms while reducing operational complexity by providing a unified interface for configuration, monitoring, and response management, rather than requiring separate monitoring systems for each cluster.
Solution Approach 2:
The remote failure detect module is designed with universal functionality to monitor various cluster types, node states, and failure conditions through a single platform. This multi-functional design achieves reliable detection across diverse scenarios without increasing complexity, as the same core monitoring engine handles different monitoring tasks through configurable parameters rather than requiring separate specialized systems.
Data Source
AI summary
Direct monitoring of a plurality of storage nodes in a primary cluster is performed based on connectivity with the storage nodes. Indirect monitoring of a first storage node is performed, in response to direct monitoring of the first storage node indicating failure of the connectivity with the first storage node, wherein a second storage node of the plurality of nodes is a backup node for the first storage node. The indirect monitor of the first storage node indicates failure of the first storage node in response to performance of storage access operations by the second storage node that were previously performed by the first storage node. A cluster-switch operation is initiated to switch to from the primary cluster to a backup cluster based on an occurrence of at least one cluster-failure condition that comprises the indirect monitor of the first storage node indicating failure of the first storage node.