Multi-Instance Software Upgrades with Ordered Zone Node Acquisition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Upgrading multiple instances of distributed software across computing zones in a cloud environment often leads to deadlock due to contention for scarce computing nodes, especially when using blue-green deployment methods, resulting in resource scarcity and potential downtime.
Innovation Solution
Enforce a common computing zone order for node acquisition during the upgrade process, ensuring that nodes are acquired in a specified sequence to prevent deadlock, allowing for partial parallelization of upgrade processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If multiple instances perform upgrades in parallel across computing zones, then upgrade speed is improved, but deadlock occurs due to node contention
Solution Approach 1:
The patent segments the upgrade process into distinct phases: node acquisition phase and service migration phase. During node acquisition, instances follow a predetermined computing zone order to acquire nodes sequentially rather than simultaneously, preventing deadlock. After node acquisition, instances can proceed to service migration in parallel. This segmentation resolves the contradiction by separating the contentious resource acquisition from the service migration operations.
Solution Approach 2:
The patent implements preliminary action by establishing a predetermined computing zone order before the upgrade process begins. Instances must acquire nodes in this predefined order (e.g., zone A before zone B before zone C) rather than attempting to acquire nodes in any order. This preliminary ordering prevents circular wait conditions and deadlock during the node acquisition phase, enabling safe parallel execution of subsequent migration operations.
2Productivity
If nodes are acquired simultaneously by multiple instances, then resource utilization is optimized, but deadlock results from contention
Solution Approach 1:
The patent segments node acquisition into two distinct phases: first, instances acquire nodes in a predetermined computing zone order to establish their upgrade targets; second, instances migrate services to their acquired nodes in parallel. This segmentation allows efficient resource utilization during the migration phase while preventing deadlock during the acquisition phase through the ordered approach.
Solution Approach 2:
The patent introduces a coordination mechanism that acts as an intermediary between multiple instances during node acquisition. This mediator ensures that instances follow the predetermined computing zone order when acquiring nodes, preventing simultaneous contention for the same nodes. The intermediary coordinates the acquisition process to eliminate deadlock while maintaining productivity through structured parallel execution during the migration phase.
3Reliability
If strict ordering is enforced for node acquisition, then deadlock is prevented, but upgrade time increases
Solution Approach 1:
The patent segments the upgrade process into node acquisition and service migration phases. The strict ordering is enforced only during node acquisition to prevent deadlock, while service migration can proceed in parallel without ordering constraints. This segmentation minimizes the impact of ordering requirements on overall upgrade duration by limiting the ordered operation to only the necessary acquisition phase.
Solution Approach 2:
The patent applies partial ordering - strict computing zone order is enforced only for node acquisition, not for the entire upgrade process. Once nodes are acquired, instances can migrate services in any parallel order without causing deadlock. This partial application of ordering constraints reduces the time penalty while maintaining deadlock prevention where it is most critical.
Data Source
AI summary
Techniques for preventing deadlock when upgrading a plurality of instances of a software service that is distributed across multiple different computing zones. Upgrade software executing on a cloud computer system receives an upgrade request to upgrade the plurality of instances. Respective upgrade processes are initiated in parallel. Node acquisition portions of the respective upgrade processes have a constraint on parallelization, as they are performed using a common upgrade procedure in which a given instance is upgraded by acquiring nodes in different ones of the computing zones according to a specified order. After acquiring the nodes according to the specified order, an updated instance is deployed to the acquired nodes to update the given instance. The acquiring of the nodes may be performed by node-securing pods in some embodiments, with the specified order enforced with affinity and anti-affinity rules.