Shared Storage Failure Recovery via Messaging Protocol Consensus
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems managing multiple nodes face challenges in efficiently handling storage device failures, particularly in shared storage architectures, where node-to-node communication can be intermittent, leading to inconsistent failure detection and recovery processes, risking catastrophic failures.
Innovation Solution
A messaging protocol is implemented to facilitate consensus among nodes regarding the state of shared storage, enabling coordinated recovery from storage device failures, including localized adapter failures, by using fault disk registries and a cluster interconnection layer to synchronize information and manage disk ownership through Software-Based Disk Ownership (SANOWN) mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single master node manages storage devices in a storage system, then the system can simplify failure handling procedures, but the system risks catastrophic failure when the master node fails or becomes unavailable
Solution Approach 1:
The patent segments the master node's failure handling responsibilities by introducing multiple coordinator nodes that can independently detect and respond to storage device failures. Each coordinator node maintains local knowledge of storage device states, allowing the system to distribute the coordination function rather than relying on a single master node. This segmentation enables failure handling to continue even when the original master node becomes unavailable.
Solution Approach 2:
The patent implements preliminary action by having coordinator nodes proactively monitor storage device states and maintain up-to-date information about device health. When a storage device fails, the coordinator nodes are already positioned and informed to immediately initiate recovery procedures without waiting for master node intervention. This preliminary preparation of coordinator nodes ensures continuous system operation during failures.
2Speed
If storage device failures are handled independently by individual nodes, then the system can improve response speed, but the system risks inconsistent failure detection and recovery processes
Solution Approach 1:
The patent implements feedback mechanisms where coordinator nodes exchange information about storage device states and recovery progress. When a coordinator node detects a failure or initiates recovery, it communicates this information to other coordinator nodes, which can then adjust their actions accordingly. This feedback loop ensures that all nodes maintain consistent understanding of the storage system state while allowing rapid local response to failures.
Solution Approach 2:
The patent merges the independent failure detection capabilities of individual nodes with coordinated recovery actions. Multiple coordinator nodes simultaneously monitor storage devices and can independently detect failures, but they combine their efforts through information sharing to execute unified recovery procedures. This merging approach preserves the speed benefits of independent detection while achieving the consistency of coordinated response.
Data Source
AI summary
The disclosed embodiments relate to systems and methods for coordinating management of a shared disk storage between nodes. Particularly, a messaging protocol may be used to communicate notifications regarding each node's perception of the shared storage's state. The nodes may use the messaging protocol to achieve consensus when recovering from a storage device failure. Some embodiments provide for recovery when localized failures, such as failures at an adapter on a node, occur.