Node Upgrade Ordering to Minimize Leader Elections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Upgrading computing nodes in a cloud infrastructure can trigger an undesirable number of leader elections, particularly in leader-follower topologies, if not managed with a selective upgrade order, leading to service disruptions.
Innovation Solution
An image upgrade controller determines an upgrade order for nodes based on ranks assigned by a ranking service, prioritizing nodes with higher-ranked pods or applications to minimize leader elections and service disruptions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If nodes are upgraded without a selective order, then the upgrade process is simple and fast, but the number of leader elections increases causing service disruptions
Solution Approach 1:
The system performs preliminary actions by determining the upgrade order before actually upgrading nodes. The upgrade controller identifies which nodes should be upgraded first based on their current state and role in the distributed system, ensuring that leader elections are minimized before they can cause service disruptions. This preliminary planning prevents the contradiction by establishing a safe upgrade sequence in advance.
Solution Approach 2:
The upgrade process is segmented into distinct phases and node groups. Instead of upgrading all nodes simultaneously or in a fixed sequence, the system divides nodes into segments based on their roles (leader/follower) and upgrades them in controlled batches. This segmentation allows the system to maintain service continuity by ensuring that at least some follower nodes remain operational during the upgrade process.
2Ease of operation
If nodes are upgraded in a fixed order, then the upgrade process is manageable, but service disruptions occur due to leader elections
Solution Approach 1:
The upgrade order is made dynamic rather than fixed. The system continuously monitors the state of nodes and adjusts the upgrade sequence based on current conditions, such as which nodes are currently leaders or followers. This dynamic approach allows the upgrade controller to adapt to changing system states and avoid triggering unnecessary leader elections, thereby maintaining service continuity while keeping the upgrade process manageable.
Solution Approach 2:
The system changes key parameters during the upgrade process, specifically the roles of nodes (leader/follower status) and the upgrade state of each node. By carefully controlling when and how these parameters change, the system can perform upgrades without causing disruptive leader elections. The upgrade controller manipulates these parameters in a sequence that maintains service continuity while achieving the upgrade goal.
3Adaptability or versatility
If leader elections are allowed to occur frequently, then the system can adapt to node failures, but service disruptions increase during upgrades
Solution Approach 1:
The system applies preliminary anti-action by preventing leader elections before they can occur during the upgrade process. The upgrade controller identifies potential leader elections in advance and adjusts the upgrade sequence to avoid them. By taking this preventive measure, the system maintains its adaptability to node failures while avoiding service disruptions that would result from frequent leader elections during upgrades.
Data Source
AI summary
Techniques are disclosed relating to upgrading nodes. A computer system may detect that an updated node image has been created that is an update to a previous node image used to deploy a plurality of nodes into an environment associated with the computer system. The computer system may determine an upgrade order in which to upgrade a set of the plurality of nodes. The upgrade order may be determined based on a rank associated with a given node of the set of nodes by a ranking service. The computer system may then upgrade the set of nodes according to the upgrade order. The upgrading of a first node of the set of nodes may include deploying a second node based on the updated node image and then evicting a set of application pods from the first node and redeploying the set of application pods onto the second node.