Durable Handle Failover for Distributed File Server Upgrades
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file servers face challenges in maintaining seamless connectivity during planned outages, such as upgrades, without incurring performance penalties associated with continuous availability features like synchronous write operations.
Innovation Solution
Implementing durable handles stored in persistent volume groups that are transferred during failover events, allowing asynchronous writes to be synchronized before planned failures, and maintaining lock states to ensure seamless reconnects without continuous availability features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous write operations are implemented for continuous availability, then reliability is improved, but productivity deteriorates due to performance penalties
Solution Approach 1:
The system performs preliminary actions by flushing asynchronous write operations to stable storage before a planned outage occurs. This ensures data is persisted in advance, allowing the system to maintain reliability during failover without requiring synchronous writes during normal operation, thus avoiding performance penalties.
Solution Approach 2:
The system dynamically switches write operation modes based on system state. During normal operation, asynchronous writes are used for optimal performance. Before a planned outage, the system transitions to flushing writes to stable storage. During failover, the target computing node restores durable handles and continues operation, dynamically adapting to maintain both reliability and productivity.
2Adaptability or versatility
If computing nodes are taken down for upgrades, then adaptability is improved, but reliability worsens due to potential service disruption
Solution Approach 1:
Before taking a computing node down for upgrade, the system performs preliminary actions by flushing all asynchronous write operations to stable storage and transferring durable handles to a target computing node. This ensures that when the source node is taken offline, service continuity is maintained through the target node, eliminating reliability concerns during upgrades.
Solution Approach 2:
The system uses a target computing node as an intermediary during the upgrade process. The target node receives and restores durable handles from the source node's persistent volume group, acting as a mediator that maintains service continuity while the source node undergoes upgrade, thus resolving the contradiction between adaptability and reliability.
3Speed
If durable handles are stored in memory for fast access, then speed is improved, but reliability deteriorates during outages
Solution Approach 1:
The system performs preliminary flushing of durable handles from memory to the persistent volume group before outages occur. This ensures that even though handles are stored in memory for fast access during normal operation, they are persisted to stable storage in advance, maintaining reliability during outages without sacrificing speed during normal operation.
Solution Approach 2:
The system creates a copy of durable handles in the persistent volume group from the in-memory storage. This copying mechanism allows the system to maintain fast in-memory access during normal operation while ensuring persistence through the copied version on stable storage, resolving the contradiction between speed and reliability.
Data Source
AI summary
Examples of systems described herein may manage durable handles for seamless failover. Durable handles may be stored in a persistent volume group. The persistent volume group may be moved, together with all or a portion of file share data, to another computing node responsive to an indication of a failure event. To accommodate asynchronous write shares, asynchronous write operations may also be provided to the target computing node. The asynchronous write operations may be processed before reconnecting clients to the file share.