Autonomous I/O Ingestion and Data Flushing in Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, ensuring that multiple nodes associate data with the correct flushing work sets (FWS) is challenging, leading to latency due to dependency on a primary node for coordinating FWS arrangement and data flushing.
Innovation Solution
Nodes in the storage system autonomously ingest I/O requests and flush data to storage by determining sequence separators in a page descriptor ring, allocating sequence identification numbers, and managing counters for pending I/O requests to ensure consistent data association with FWSs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a primary node coordinates FWS arrangement and data flushing for multiple nodes, then data consistency is ensured, but latency increases due to dependency on the primary node
Solution Approach 1:
The system segments the centralized coordination function into distributed autonomous operations. Each node independently determines sequence separators and manages its own FWS, eliminating the single-point coordination bottleneck while maintaining consistency through shared sequence ID namespace and separator synchronization mechanisms
Solution Approach 2:
Nodes perform preliminary actions by pre-determining sequence separators and pre-allocating sequence ID numbers before I/O requests arrive. This allows nodes to be ready to immediately process and flush data without waiting for primary node coordination, reducing latency while ensuring data consistency
2Productivity
If nodes autonomously process I/O requests in parallel, then system performance improves, but difficulty in ensuring correct FWS association increases
Solution Approach 1:
The patent introduces sequence ID numbers and sequence separators as intermediary mechanisms that mediate between autonomous nodes and FWS management. These intermediaries provide a standardized interface that simplifies autonomous operation while ensuring correct FWS association through mathematical relationships (sequence ID modulo number of FWS)
Solution Approach 2:
The system uses parameter changes in the form of sequence ID numbers and sequence separators to dynamically manage FWS association. By changing these numerical parameters based on predetermined rules, nodes can autonomously determine correct FWS without complex coordination logic, enabling parallel processing while maintaining correctness
3Reliability
If nodes wait for primary node coordination before flushing data, then data consistency is maintained, but I/O request processing is interrupted
Solution Approach 1:
Nodes perform self-service by autonomously monitoring their own pending I/O request counters and independently deciding when to flush data based on local state. Each node services itself without requiring primary node intervention, maintaining consistency through shared sequence separator knowledge while eliminating processing interruptions
Solution Approach 2:
The system implements feedback mechanisms where nodes continuously monitor their pending I/O request counters and use this feedback to autonomously trigger flush operations. This local feedback loop enables nodes to maintain data consistency while avoiding interruptions, as each node adjusts its flushing behavior based on its own real-time state
Data Source
AI summary
Nodes in a storage system can autonomously ingest I/O requests and flush data to storage. First and second nodes determine a sequence separator, the sequence separator corresponding to an entry in a page descriptor ring that separates two flushing work sets (FWS). The first node receives an input/output (I/O) request and allocates a sequence identification (ID) number to the I/O request. The first node determines a FWS for the I/O request based on the sequence separator and the sequence ID number, and commits the I/O request using the sequence ID number. The I/O request and the sequence ID number are sent to the second node.