Dirty-Node Write Buffer Clearing for Cluster Storage Volumes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In container orchestration platforms like Kubernetes, when a node fails non-gracefully, residual data in write buffers can lead to data corruption in storage volumes due to incomplete or unsynchronized data writes, as the scheduler cannot determine the node's status and reassign tasks effectively.
Innovation Solution
A storage orchestrator controller marks nodes as 'dirty' upon failure, preventing data writes by erasing buffers and denying volume access until the node is cleaned, ensuring data integrity by only allowing clean nodes to mount volumes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the node is allowed to continue operating after failure, then system availability is improved, but data integrity deteriorates due to residual data in write buffers
Solution Approach 1:
The system performs preliminary actions by marking the node as dirty upon failure detection, preventing any subsequent data writes to storage volumes until the buffer is cleared. This proactive measure ensures that even if the node recovers and attempts to write residual data, the marking mechanism blocks it, thus preserving data integrity while allowing the node to remain available for other operations.
Solution Approach 2:
The patent introduces an intermediary mechanism (the dirty marking system) that mediates between the failed node and the storage volumes. This intermediary layer prevents direct interaction between the node's write buffers and the storage volumes, blocking potential data corruption while allowing the node to recover and rejoin the cluster safely.
2Reliability
If the write buffer is erased immediately upon failure, then data integrity is improved, but loss of information occurs for data that may have been successfully written
Solution Approach 1:
The system dynamically adjusts the state of the node based on its operational status. The dirty marking is not permanent but conditional - it can be cleared when the node successfully reconnects and synchronizes with the cluster. This dynamic approach allows the system to preserve data integrity during failure while recovering any potentially valid data writes once the node is confirmed to be operational again.
Solution Approach 2:
The patent changes the operational parameter of the node from 'clean' to 'dirty' upon failure, which fundamentally alters its interaction with storage volumes. This parameter change acts as a flag that prevents writes without physically erasing buffer content immediately. The parameter can be reversed when the node recovers, allowing for potential data recovery while maintaining integrity during the failure state.
3Reliability
If the node is marked as dirty and access is denied, then data corruption is prevented, but system productivity decreases due to restricted node access
Solution Approach 1:
The patent segments the node's access rights to storage volumes based on its operational state. Instead of completely disabling the node, only the data write function is segmented and blocked when the node is marked as dirty. The node can still perform other cluster functions, and storage volumes can be accessed by other clean nodes, thus maintaining overall system productivity while protecting data integrity.
Data Source
AI summary
The technology disclosed herein enables a storage orchestrator controller to prevent residual data from being written to a storage volume when a node fails non-gracefully. In a particular example, a method includes determining a health status of nodes in the cluster and, in response to determining a node in the cluster failed, marking the node as dirty. After marking the node as dirty and in response to determining the node is ready, the method includes directing the node to erase data in one or more write buffers at the node. The one of more write buffers buffer data for writing to one or more storage volumes when the one or more storage volumes are mounted by the node. After the one or more write buffers are erased, the method includes marking the node as clean.