Container Workload Recovery via Node Failure Isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing container management systems, such as Kubernetes, lack effective techniques for managing node failures, particularly in scenarios where a node may be down or experiencing communication issues, leading to uncertainty and prolonged recovery times for stateful applications.
Innovation Solution
The proposed solution involves a method to detect node failures, isolate the failed node, delete configuration objects, and create replacement containerized workloads on healthy nodes, using a podmon architecture that interacts with the Kubernetes API and CSI driver to manage persistent storage and node states, ensuring timely restoration of stateful applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system waits for node recovery before taking action, then false positives are avoided, but recovery time is prolonged
Solution Approach 1:
The system performs preliminary actions by immediately isolating storage volumes and marking configuration objects when node failure is detected, rather than waiting for full recovery confirmation. This preliminary isolation prevents data corruption while the node is recovering, and allows replacement workloads to be scheduled sooner on healthy nodes.
Solution Approach 2:
The system applies preliminary anti-action by proactively preventing new workload scheduling on failed nodes through configuration object marking, and isolating storage volumes before the node can potentially cause data corruption. This counter-measure is taken in advance to prevent the harmful effect of data corruption that might occur if the node recovers in an inconsistent state.
2Reliability
If storage volumes are isolated immediately upon node failure detection, then data corruption is prevented, but system resources are tied up longer
Solution Approach 1:
The system extracts the storage volumes from the failed node immediately upon detection, isolating them from the node to prevent data corruption. This extraction is performed separately from the node itself, allowing the storage volumes to be protected while the node is being recovered or replaced, thus maintaining data integrity without permanently tying up resources.
Solution Approach 2:
The system discards the association between storage volumes and the failed node, marking configuration objects to prevent new workloads from being scheduled on the failed node. When the node recovers or is replaced, the storage volumes can be recovered and reassigned to new workloads on healthy nodes, thus preventing permanent resource loss.
3Measurement precision
If configuration objects are marked to prevent scheduling on failed nodes, then workload placement accuracy is improved, but scheduling flexibility is reduced
Solution Approach 1:
The system applies dynamic marking to configuration objects, where the failed node indicator is added temporarily upon failure detection and can be removed when the node recovers or is properly replaced. This dynamic approach maintains precise failure state identification while allowing scheduling flexibility to be restored once the node is no longer in a failed state, thus adapting to changing system conditions.
Data Source
AI summary
Techniques for managing node failures in container environments are disclosed. In one example, a method determines when a first node executing at least one containerized workload has failed. In response to determining the first node has failed, the method marks a configuration object for the first node with an indicator that the first node is not to be used to schedule execution of a subsequent containerized workload, isolates from the first node one or more storage volumes used by the first node, and deletes configuration objects for the one or more storage volumes and for the containerized workload. The method then causes creation of a replacement containerized workload for execution on a second node, removes one or more artifacts associated with the containerized workload from the first node, and removes the indicator from the configuration object for the first node.