Prechecking for Non-Disruptive Data Management System Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data management systems face challenges in performing non-disruptive updates, as they can fail due to changes in the state of storage nodes during the update process, leading to prolonged downtime and potential system errors, especially when initial prechecks indicate readiness but subsequent node-level checks reveal issues like storage capacity or network configuration changes.
Innovation Solution
Implementing a dual prechecking process that includes both cluster-level and node-level checks before initiating a non-disruptive update, allowing for the selection between disruptive and non-disruptive update procedures based on precheck results, and performing individual node-level prechecks during the rolling update to ensure each storage node is still in a state for successful update.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a non-disruptive update procedure is used to update storage nodes one-at-a-time, then system availability is maintained during the update, but the update process becomes more complex and time-consuming
Solution Approach 1:
The update procedure is segmented into distinct phases: initial cluster-level prechecking, rolling update execution with individual node-level prechecks, and post-update validation. Each storage node is updated individually rather than in parallel, dividing the update process into manageable segments that maintain system availability while reducing overall complexity through structured progression
Solution Approach 2:
Cluster-level prechecking is performed before initiating the rolling update to verify system readiness. Additionally, node-level prechecks are conducted immediately before updating each individual storage node during the rolling update process. These preliminary actions identify and address potential issues beforehand, preventing update failures without requiring complex rollback mechanisms
2Measurement precision
If cluster-level prechecking is performed before initiating update, then update readiness can be assessed, but issues at individual node level may be missed leading to update failures
Solution Approach 1:
The prechecking process is segmented into two levels: cluster-level prechecking that assesses overall system readiness before update initiation, and node-level prechecks that verify individual storage node readiness immediately before each node is updated during the rolling update. This segmentation ensures comprehensive assessment at both system and component levels, preventing update failures from missed node-specific issues
Solution Approach 2:
The system implements feedback mechanisms where node-level precheck results are obtained and evaluated before proceeding with each node update. If node-level prechecks fail, the rolling update process can be adjusted or halted for that specific node, providing feedback that prevents update failures while maintaining overall update progression for other nodes
3Reliability
If node-level prechecks are performed during rolling update for each storage node, then update failures are reduced, but the overall update time increases
Solution Approach 1:
Node-level prechecks are performed periodically at specific intervals during the rolling update process, specifically immediately before updating each individual storage node. This periodic checking ensures reliability by catching node-specific issues before they cause update failures, while the structured timing prevents excessive delays by only checking at critical decision points rather than continuously
Solution Approach 2:
The rolling update process maintains continuity by updating storage nodes one-at-a-time while other nodes remain operational and serving the system. This continuous operation allows the update process to progress without complete system shutdown, reducing overall downtime despite the added node-level prechecks, as the useful action of data service continues uninterrupted on non-updated nodes
Data Source
AI summary
Aspects of data management are described. Information for updating a cluster of storage nodes to a second version may be obtained. Based on obtaining the information, a cluster-level precheck procedure may be performed prior to initiating an update procedure for the cluster of storage nodes. Also, during the update procedure, a node-level precheck procedure may be performed prior to initiating an update of a storage node to determine whether the storage node is in a state for successful update. Based on a result of the node-level precheck, a determination of whether to delay the update of the storage node may be made.