Preemptible Node Graceful Shutdown via Early Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud service providers do not provide customers with notice of node preemption, leading to drawbacks in handling node preemption, as current techniques violate service level agreements when attempting to delay shutdowns.
Innovation Solution
Implementing a method that includes generating and labeling preemptible nodes with a specific capacity type, deploying containerized applications that detect preemption, and executing shutdown operations using a cluster management service, allowing for graceful shutdowns without violating service level agreements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If preemptible nodes delay shutdown to execute wind-down procedures, then workload completion improves, but service level agreements are violated
Solution Approach 1:
The system performs preliminary actions by detecting preemption events early and initiating shutdown procedures before the actual preemption occurs. The cluster management service monitors preemptible nodes and triggers wind-down operations in advance, allowing workloads to complete gracefully without violating service level agreements. This preliminary detection and response mechanism resolves the contradiction by enabling timely shutdown initiation that balances workload completion with adherence to SLA time constraints.
2Adaptability or versatility
If preemptible nodes are evicted without notice, then cloud service provider capacity flexibility improves, but customer workload stability deteriorates
Solution Approach 1:
The system implements feedback mechanisms where the cluster management service continuously monitors preemptible nodes for preemption events. When preemption is detected, the system provides feedback by triggering shutdown procedures that notify customers and initiate graceful workload termination. This feedback loop maintains capacity flexibility while improving workload stability through proactive detection and structured response to preemption events.
Data Source
AI summary
Conventional techniques for shutting down preempted nodes includes drawbacks to cloud users and service providers alike. The disclosed techniques are directed to mitigating or eliminating these drawbacks. Upon receiving a preemptible node request, a preemptible node may be generated, labeled as having a particular capacity type, and added to a cluster managed by a cluster manager. In response to detecting the label, the cluster manager may deploy a containerized application to the preemptible node. The containerized application may monitor node metadata to detect preemption of the node. Node metadata may be provided by node metadata service executing at a smart network interface card connected to a host on which the preemptible node executes. In response to detecting preemption, the containerized application may initiate shutdown and/or replacement operations of the preemptible node to reduce or eliminate the negative impact of preemption.