Storage Node Quiescence for Non-Disruptive Cluster Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in performing software updates without disrupting the services provided by the storage cluster, leading to job failures and prolonged downtime during the update process.
Innovation Solution
Implementing a quiescent operation where storage nodes enter a quiescent state before updating, allowing them to execute in-process jobs while refraining from new ones, and using overlapping update threads to manage the update process across nodes, thereby reducing job failures and shortening the update duration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If storage nodes update software immediately without quiescing, then update speed is fast, but job failures increase and service reliability deteriorates
Solution Approach 1:
The system enters a quiescent state before performing software updates to allow in-process jobs to complete. This preliminary action prevents job failures during updates while maintaining relatively fast update speeds by not requiring full service shutdown.
2Reliability
If storage nodes perform quiescent operation before updating, then job failures are reduced and reliability improves, but update duration increases
Solution Approach 1:
The system performs a partial quiescent operation that allows in-process jobs to complete while preventing new jobs from being assigned. This partial approach provides sufficient reliability improvement without requiring a complete system shutdown, thereby limiting the increase in update duration.
3Productivity
If storage nodes continue executing new jobs during update, then productivity is maintained, but job execution stability deteriorates causing failures
Solution Approach 1:
The system preemptively stops accepting new jobs before the software update begins, while allowing currently in-process jobs to complete. This preliminary action maintains productivity for ongoing work while ensuring execution stability during the update transition.
Data Source
AI summary
Aspects of data management are described. During an update procedure for serially updating a cluster of storage nodes, a storage node of the cluster of storage nodes may enter a quiescent state. While in the quiescent state, the storage node may refrain from obtaining new jobs and may continue to execute jobs that were initiated at the storage node prior to entering the quiescent state. The storage node may enter the quiescent state while another storage node enters an update state for installing the update version. The storage node may also post, to a job queue, jobs running at the storage node that are terminated at an end of the quiescent state.