Upgrade Controller Locking for Node and Pod Availability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of coordinating upgrades in cloud infrastructure to prevent data unavailability and service disruptions when multiple upgrade controllers operate independently, particularly in scenarios where data is replicated across multiple nodes, is addressed.
Innovation Solution
Implementing a locking mechanism using a lock store and lock API to synchronize and coordinate upgrades by pod and node upgrade controllers, ensuring exclusive access to node pools during upgrades, thereby preventing concurrent upgrades that could lead to data unavailability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple upgrade controllers operate independently to upgrade nodes and pods, then upgrade flexibility and parallelism are improved, but service disruption and data unavailability increase
Solution Approach 1:
A lock store is introduced as an intermediary component between multiple upgrade controllers and the node pool. The lock store maintains lock records that coordinate upgrades by allowing one controller to acquire exclusive access (lock) on the node pool, preventing other controllers from simultaneously upgrading the same resources. This mediator enables parallel upgrade operations across different node pools while ensuring sequential, conflict-free upgrades within each node pool, thus maintaining service availability without sacrificing overall upgrade productivity.
2Speed
If upgrades are performed without coordination, then upgrade speed is improved, but data unavailability and service disruptions worsen
Solution Approach 1:
Before an upgrade controller begins upgrading nodes or pods in a node pool, it performs a preliminary action by acquiring a lock on that node pool through the lock store. This lock acquisition happens in advance, ensuring exclusive access rights are established before any upgrade operations commence. The preliminary locking action prevents other controllers from initiating conflicting upgrades, thereby maintaining fast upgrade execution while eliminating data unavailability caused by uncoordinated concurrent upgrades.
3Reliability
If a locking mechanism is implemented to coordinate upgrades, then service availability is improved, but system complexity increases
Solution Approach 1:
The locking and coordination logic is extracted from the upgrade controllers themselves and placed into a separate, dedicated lock store component. This extraction allows the upgrade controllers to remain simple and focused on their primary upgrade functions, while the complex lock management (acquiring, releasing, and coordinating locks) is handled by the specialized lock store. The lock store communicates through simple lock records and API calls, providing sophisticated coordination functionality without increasing the complexity of the upgrade controllers themselves, thus maintaining service availability while managing system complexity.
Data Source
AI summary
Techniques are disclosed pertaining to performing upgrades by upgrade controllers. A node upgrade controller determines to upgrade a first set of nodes, of a plurality of nodes, on which executes a first set of pods that facilitate access to an instance of a particular component. The node upgrade controller obtains a lock on the plurality of nodes to prevent a pod upgrade controller from upgrading a second set of pods that execute on a second set of the plurality of nodes and facilitate access to another instance of the particular component. The unavailability of the first and second sets of pods causes a requisite number of instances of the particular component to be unavailable. The node upgrade controller upgrades the first set of nodes and releases the lock to allow the pod upgrade controller to obtain a lock on the plurality of nodes.