Cluster Node Buffer Erasure After Non-Graceful Failure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In container orchestration platforms like Kubernetes, non-graceful node failures can lead to residual data writes, causing data corruption in storage volumes due to incomplete or pending data transfers when nodes become unavailable.
Innovation Solution
A storage orchestrator controller marks nodes as 'dirty' upon failure, preventing data writes by erasing buffers and denying volume mounting requests until the node is cleaned, ensuring data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a node fails non-gracefully (loses network connection, crashes, loses power), then the node becomes unavailable, but residual data in write buffers may be written to storage volumes causing data corruption
Solution Approach 1:
The system performs preliminary actions by marking the node as dirty upon failure detection, which proactively prevents any subsequent write operations from dirty nodes before data corruption can occur. This preliminary marking mechanism ensures that even if the node recovers and attempts to write buffered data, the storage system will reject the write operation.
Solution Approach 2:
The system applies preliminary anti-action by erasing the write buffer contents when a dirty node attempts to mount a storage volume. This counter-action directly neutralizes the harmful residual data before it can be written to the storage volume, preventing data corruption while allowing the node to recover and be remounted.
2Reliability
If the storage orchestrator denies mounting requests from dirty nodes, then data corruption is prevented, but node access to storage volumes is blocked until the node is cleaned
Solution Approach 1:
The system dynamically changes the node's mounting status based on its cleanliness state. Nodes transition between dirty and clean states, with their ability to mount storage volumes dynamically adjusted accordingly. This dynamic mechanism allows the system to automatically restore node access once the node is cleaned, balancing data protection with operational flexibility.
3Reliability
If the system erases write buffers on dirty nodes, then residual data is removed preventing corruption, but additional operations are required increasing system complexity
Solution Approach 1:
The system implements self-service by automatically detecting node failures, marking nodes as dirty, and erasing their write buffers without requiring manual intervention. The storage orchestrator continuously monitors node health and autonomously executes the necessary cleanup operations, reducing the need for complex manual failure handling procedures.
Data Source
AI summary
The technology disclosed herein enables a storage orchestrator controller to prevent residual data from being written to a storage volume when a node fails non-gracefully. In a particular example, a method includes determining a health status of nodes in the cluster and, in response to determining a node in the cluster failed, marking the node as dirty. After marking the node as dirty and in response to determining the node is ready, the method includes directing the node to erase data in one or more write buffers at the node. The one of more write buffers buffer data for writing to one or more storage volumes when the one or more storage volumes are mounted by the node. After the one or more write buffers are erased, the method includes marking the node as clean.