Cloud Service Healing During Upgrades
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud computing systems face challenges in migrating services away from faulted nodes and handling faults during upgrades, introducing downtime and potential data loss due to multiple points of failure in distributed applications.
Innovation Solution
A computer system determines faulted virtual machines and reallocates their resources to a different cloud node, re-instantiating them while pausing and managing upgrades to ensure seamless service continuity by restoring virtual machines to either the newer or older service version based on their upgrade domain status.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed applications are deployed on multiple cloud nodes, then application availability and scalability are improved, but the number of points of failure increases
Solution Approach 1:
The system segments the distributed application into multiple independent virtual machine instances deployed across different cloud nodes. Each instance can be independently managed, monitored, and restored, allowing the system to maintain availability even when individual nodes fail. The segmentation enables granular fault isolation and targeted recovery operations.
Solution Approach 2:
The system implements prior cushioning by maintaining backup virtual machine images and configuration data before failures occur. When a node fails, the system can rapidly restore services using pre-prepared backup data, minimizing downtime. The upgrade domain mechanism also serves as a cushion by staging updates in controlled groups before full deployment.
2Reliability
If services are migrated away from faulted cloud nodes, then service continuity is improved, but migration time and complexity increase
Solution Approach 1:
The system performs preliminary actions by pre-configuring backup virtual machine images and maintaining resource allocation data before failures occur. When a node fails, the migration process leverages these pre-prepared resources to rapidly instantiate services on healthy nodes, significantly reducing migration time compared to creating services from scratch.
Solution Approach 2:
The system creates copy instances of virtual machines on healthy cloud nodes when failures are detected. Instead of complex live migration, the system rapidly copies service configurations and data from the faulted node's allocated resources to new virtual machine instances on healthy nodes, enabling quick service restoration.
3Adaptability or versatility
If virtual machines are upgraded from older to newer service versions, then service functionality is improved, but faults during upgrade may cause downtime
Solution Approach 1:
The system segments the upgrade process into multiple upgrade domains, each containing a subset of virtual machines. Upgrades are applied incrementally across domains rather than all at once, allowing the system to maintain service availability by keeping non-upgraded instances running while upgrading others. This segmentation isolates upgrade-related faults to specific domains.
Solution Approach 2:
The system implements beforehand cushioning by maintaining both older and newer service version capabilities during the upgrade transition. Virtual machines can fall back to the older service version if faults occur during upgrade, ensuring continuous availability. The system cushions against upgrade failures by preserving the ability to run previous versions alongside new versions.
Data Source
AI summary
Embodiments described herein are directed to migrating affected services away from a faulted cloud node and to handling faults during an upgrade. In one scenario, a computer system determines that virtual machines running on a first cloud node are in a faulted state. The computer system determines which cloud resources on the first cloud node were allocated to the faulted virtual machine, allocates the determined cloud resources of the first cloud node to a second, different cloud node and re-instantiates the faulted virtual machine on the second, different cloud node using the allocated cloud resources.