Fencing Failed Nodes in Multi-Node File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-node file systems, fencing a failed node to prevent I/O operations when the file system manager lacks direct access to all disks is challenging, leading to potential data corruption and application failures, especially in heterogeneous environments.
Innovation Solution
A method where the filesystem manager attempts to fence all disks from the failed node and, if unsuccessful, sends the fencing command to servers with access to those disks, iteratively forwarding the command until access is limited across all disks in the subsystem, ensuring recovery and preventing data corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the file system manager directly issues fencing calls to the disk subsystem, then the fencing operation is simple and direct, but it fails when the file system manager does not have access to all disks due to path failure or heterogeneous environments
Solution Approach 1:
The patent introduces intermediary nodes (other servers in the cluster) that have direct access to the disk subsystem. When the file system manager cannot directly fence a disk, it sends the fencing command through these intermediary nodes, which then execute the fencing operation locally. This mediator approach resolves the access problem without requiring direct file system manager access to all disks.
Solution Approach 2:
The patent segments the fencing operation into multiple independent paths. Instead of requiring a single direct path from the file system manager to all disks, the fencing function is divided and distributed across multiple nodes in the cluster. Each node handles fencing for the disks it can access, ensuring complete coverage through segmented operations.
2Reliability
If the file system manager sends fencing commands to multiple servers iteratively, then complete fencing coverage is achieved, but the fencing process time increases
Solution Approach 1:
The patent maintains a pre-established mapping between disks and the servers that have access to them. This preliminary organization of access information allows the file system manager to quickly determine which servers can fence which disks without iterative discovery, significantly reducing the time required to initiate and complete the fencing process across multiple nodes.
Data Source
AI summary
In a multi-node computer system, file access by a failed node is limited. Upon receipt on an indication of a node failure, a fencing command is sent to disks in a disk subsystem to which the failed node has access. If the fencing command sent to a disk fails, the fencing command is sent to a server having access to at least one disk in a disk subsystem to which the failed node has access to limit access by the failed node to the disk in the disk subsystem. If the fencing command sent to the server does not result in limiting access by the failed node to all the disks in the disk subsystem, sending the command to another server having access to at least one disk in the disk subsystem to limit access by the failed node to the disks in the disk subsystem. The fencing command may be sent to various servers until access by the failed node to all the disks in the disk subsystem is limited or until the fencing command has been sent to all the servers. The fencing command may be sent one at a time to servers having access to the disks in the disk subsystem, may be sent concurrently to all the servers having access to the disks in the disk subsystem, or may be forwarded from one server to another.