Distributed File System Upgrade Coordinator for Parallel Pre-Reboot Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, rolling upgrades are inefficient due to sequential node unavailability, which limits the parallelization of upgrade operations and prolongs the upgrade process.
Innovation Solution
A data storage system with components that facilitate a rolling upgrade by initializing a first computing node to upgrade from a first file system version to a newer version, activating an upgrade coordinator to perform pre-reboot operations concurrently on other nodes, and leveraging the upgraded version's features to accelerate the upgrade process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If rolling upgrade is performed sequentially on each node, then system availability is maintained, but upgrade duration is prolonged
Solution Approach 1:
The upgrade process is segmented into two independent phases: pre-reboot operations and post-reboot operations. Multiple nodes can perform pre-reboot operations concurrently, while only one node performs post-reboot operations sequentially to maintain system availability. This segmentation resolves the contradiction by enabling parallelization where safe and sequential execution where necessary.
Solution Approach 2:
Pre-reboot operations are performed as preliminary actions before the actual system reboot. These operations include data synchronization, metadata updates, and other preparatory tasks that can be executed in parallel across multiple nodes without affecting system availability, thereby reducing overall upgrade duration while maintaining reliability.
2Stability of the object's composition
If sequential upgrade is performed to maintain compatibility, then system stability is ensured, but productivity during upgrade is reduced
Solution Approach 1:
The upgrade workflow is divided into stability-critical sequential post-reboot operations and productivity-enhancing parallel pre-reboot operations. This segmentation allows the system to maintain stability through controlled sequential execution while improving productivity through concurrent pre-reboot operations across multiple nodes.
Solution Approach 2:
Multiple nodes perform pre-reboot operations concurrently (excessive action in terms of parallelism), which accelerates the upgrade process. The system tolerates this increased activity in the pre-reboot phase because these operations are designed to be safe and reversible, thereby improving productivity without compromising system stability.
3Productivity
If concurrent pre-reboot operations are performed on multiple nodes, then upgrade efficiency is improved, but coordination complexity increases
Solution Approach 1:
Coordination complexity is reduced by segmenting the upgrade process into two distinct phases with different coordination requirements. Pre-reboot operations use simple broadcast-based coordination where the supervisor distributes tasks to multiple nodes independently. Post-reboot operations use sequential coordination where only one node executes at a time. This segmentation manages complexity while maintaining high upgrade efficiency.
Solution Approach 2:
A supervisor node acts as an intermediary that coordinates pre-reboot operations across multiple worker nodes. The supervisor manages the concurrent execution, handles node failures, and ensures proper sequencing without requiring complex peer-to-peer coordination between worker nodes, thereby improving upgrade efficiency while controlling coordination complexity.
4Duration of action of moving object
If all nodes are upgraded simultaneously, then total upgrade time is reduced, but system availability is compromised
Solution Approach 1:
The upgrade process is segmented such that pre-reboot operations (which can be done in parallel) are separated from post-reboot operations (which require sequential execution to maintain availability). This segmentation enables simultaneous pre-reboot operations across all nodes to reduce total upgrade time, while sequential post-reboot operations ensure system availability is maintained.
Solution Approach 2:
All nodes perform pre-reboot operations simultaneously as preliminary actions before the critical post-reboot phase. This preliminary parallel execution significantly reduces total upgrade time. Only after these preliminary operations are complete does the system proceed to sequential post-reboot operations that maintain availability, thus resolving the contradiction between speed and reliability.
Data Source
AI summary
Accelerated behavior change for upgrades in a distributed system is described herein. A method as described herein can include facilitating a file system upgrade of a first computing node of a computing cluster from a first file system version to a second file system version that is newer than the first file system version, wherein the file system upgrade comprises pre-restart operations and a system restart performed subsequent to the pre-restart operations; activating a supervisor system of the first computing node in response to the first computing node completing the file system upgrade; and causing, in response to the activating, the supervisor system of the first computing node to initiate concurrent performance of the pre-restart operations of the file system upgrade at second computing nodes of the computing cluster, distinct from the first computing node.