Dynamic Fail-Over Node Selection for Rolling Patch Installations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy techniques for rolling patch installations in high-availability clustered client-server systems fail to consider a full range of criteria for selecting fail-over nodes, leading to cascading fail-over situations and increased downtime or brownouts, especially during mission-critical operations.
Innovation Solution
The method involves intelligently selecting fail-over nodes by considering various criteria such as upgrade states, version compatibility, and temporal conditions, ensuring that services are migrated to nodes that have already been upgraded, thereby reducing the number of migrations and minimizing downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If legacy fail-over techniques are used during rolling patch installation, then clients can maintain basic fail-over capabilities, but cascading fail-over situations occur and downtime increases
Solution Approach 1:
The system performs preliminary evaluation of candidate fail-over nodes before actual fail-over occurs. By assessing node status, patch application state, and compatibility in advance, the system prevents cascading fail-overs and reduces downtime by ensuring the selected node is ready to immediately assume service responsibilities.
Solution Approach 2:
The system implements feedback mechanisms that continuously monitor node status during rolling patch installation. This feedback enables dynamic adjustment of fail-over decisions based on real-time conditions, preventing selection of nodes that would cause cascading failures and minimizing service disruption.
2Ease of operation
If services are migrated to nodes that need upgrading, then fail-over can occur, but the migrated node must be upgraded causing additional migrations and increased downtime
Solution Approach 1:
The system identifies and selects fail-over nodes that have already completed their patch upgrades before initiating service migration. This preliminary selection ensures that the target node is in a stable, upgraded state and can immediately handle migrated services without requiring additional migrations, thereby reducing total downtime.
Solution Approach 2:
Instead of migrating to any available node and then upgrading it (traditional approach), the system inverts the sequence by first selecting nodes that are already upgraded, then migrating services to them. This reversal eliminates the need for post-migration upgrades and associated service disruptions.
3Productivity
If rolling patch installation is performed on all nodes simultaneously, then comprehensive system upgrade is achieved, but service availability deteriorates due to multiple concurrent migrations
Solution Approach 1:
The system segments the rolling patch installation process by dividing nodes into different upgrade phases based on their current status. By processing nodes in stages and maintaining clear separation between upgraded and non-upgraded nodes, the system enables concurrent upgrades without causing cascading fail-overs, thus maintaining service availability while achieving comprehensive system upgrade.
Solution Approach 2:
The system dynamically adjusts the fail-over selection based on the current state of the cluster during rolling patch installation. As nodes transition from non-upgraded to upgraded states, the system dynamically updates the pool of eligible fail-over nodes, ensuring optimal service availability throughout the upgrade process while maintaining high upgrade throughput.
Data Source
AI summary
A method, system, and computer program product for upgrade procedures in high-availability clustered systems. The method selects fail-over nodes during rolling patch installation and comprises steps for identifying a first node to upgrade, migrating services of the first node to a second node and shutting down the first node to perform its upgrade, then migrating the services back from the second node to the first node. When performing an upgrade of the other nodes of the cluster (e.g., nodes of the cluster other than the first node), rather than migrating the services of the node to be upgraded to another node that still needs to be upgraded, instead migrate the services of the node to be upgraded to a node that has already been upgraded. This reduces the brown-out time at least to the extent that this method incurs only one migration per node rather than two migrations per node.