Proxy Node I/O Quiesce Handling in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-node distributed storage systems, I/O quiesce and drain operations can fail to complete cluster-wide due to timing issues when I/O shipping is used, as proxy nodes may complete quiesce operations before shipping nodes, leading to incomplete acknowledgement and stalled cluster-wide states.
Innovation Solution
Allowing proxy nodes to continue receiving and processing shipped I/O requests during quiesce and drain operations ensures that shipping nodes can enter quiesce and drained states, and cluster-wide states are accurately indicated by ensuring all nodes complete their operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If proxy nodes complete quiesce operations before shipping nodes, then local quiesce state is achieved faster, but cluster-wide quiesce state cannot be completed due to incomplete acknowledgement
Solution Approach 1:
The system performs preliminary actions by having proxy nodes continue to process shipped I/O requests even after entering local quiesce state. This preliminary continuation of processing ensures that all in-flight I/O is acknowledged before the shipping node transitions to quiesced state, preventing premature state indication and ensuring cluster-wide state accuracy.
2Stability of the object's composition
If proxy nodes stop processing shipped I/O during quiesce operation, then local quiesce state is established, but shipping nodes cannot complete quiesce due to pending I/O acknowledgements
Solution Approach 1:
The system applies dynamics by allowing proxy nodes to dynamically adjust their processing behavior during quiesce operations. Proxy nodes maintain the ability to process shipped I/O requests that were already in-flight when quiesce was initiated, while blocking new local I/O. This dynamic approach enables both local quiesce state establishment and shipping node completion.
3Adaptability or versatility
If I/O shipping is used in distributed storage systems, then system flexibility and architecture are improved, but timing issues cause quiesce and drain operations to fail
Solution Approach 1:
The system implements feedback mechanisms where proxy nodes send acknowledgements to shipping nodes about the completion status of shipped I/O requests. This feedback loop ensures that shipping nodes only transition to quiesced or drained states after receiving confirmation that all their shipped I/O has been processed, thereby resolving timing issues while maintaining architectural flexibility.
Data Source
AI summary
Various systems and methods for performing Input/Output (I/O) quiesce and drain operations in multi-node distributed storage systems are disclosed. For example, one method involves receiving a message. The message indicates a request for a operation to be performed by a node, where the node can receive I/O requests from an application, as well as remote I/O requests from another node. The node can issue the I/O requests and the remote I/O requests to the one or more storage devices. In response to receipt of the message, the method performs a first portion of a operation on the node. The first portion of the operation includes the node not processing any additional locally generated I/O requests and processing additional remote I/O requests.