Data Node Fencing in Distributed File System Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems like HDFS, the single point of failure at the name node can lead to data loss and inconsistencies due to split-brain scenarios where data nodes receive commands from both active and inactive name nodes, resulting in over-replication or deletion of data blocks, especially during failover events.
Innovation Solution
Implementing data node fencing techniques that involve identifying the active name node through transaction identifiers, marking data nodes as untrusted initially, and changing their state to trusted once they acknowledge the active name node, ensuring that only the active name node issues commands and preventing commands from inactive nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high availability (HA) architecture is added to allow failover from main name node to backup name node, then system availability is improved, but data consistency deteriorates due to split-brain scenarios where data nodes receive commands from both active and inactive name nodes
Solution Approach 1:
The patent introduces an intermediary fencing mechanism that mediates between multiple name nodes and data nodes. The active name node acts as an intermediary by providing fencing tokens to data nodes, which then use these tokens to verify and filter commands from other name nodes. This intermediary token system resolves the split-brain problem by enabling data nodes to distinguish between legitimate commands from the active name node and stale commands from inactive name nodes, thus maintaining data consistency while preserving high availability.
2Reliability
If data nodes accept commands from any name node to ensure continuous operation, then system availability is improved, but data loss risk increases due to duplicate deletion commands from multiple active name nodes
Solution Approach 1:
The patent implements a feedback mechanism where data nodes send their current state and fencing token information back to name nodes. When a name node receives feedback indicating that a data node has already processed a command or has a newer fencing token, it aborts the duplicate command. This feedback loop prevents data loss by ensuring that only the active name node's commands are executed, while maintaining system availability by allowing data nodes to continue operating and providing feedback.
3Productivity
If data nodes immediately execute commands from name nodes to ensure real-time responsiveness, then operational speed is improved, but data integrity deteriorates when commands arrive after failover from inactive name nodes
Solution Approach 1:
The patent applies preliminary action by having the active name node distribute fencing tokens to all data nodes before executing commands. Data nodes use these pre-distributed tokens to validate incoming commands, checking whether the commanding name node is currently active. This preliminary token distribution enables data nodes to quickly validate commands without delaying execution, thus maintaining operational speed while ensuring data integrity by blocking stale commands from inactive name nodes.
Data Source
AI summary
Systems and methods for data node fencing in a distributed file system to prevent data inconsistencies and corruptions are disclosed. An embodiment includes implementing a protocol whereby data nodes detect a failover and determine an active name node based on transaction identifiers associated with transaction requests. The data nodes also provide to the active name node block location information and an acknowledgment. The embodiment further includes a protocol whereby a name node refrains from issuing invalidation requests to the data nodes until the name node receives acknowledgments from all data nodes that are functional.