Distributed Storage I/O Replay for Fast Failure Resumption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cross-site distributed storage systems experience delays in resuming input/output operations due to connectivity issues and failures, leading to disruptions in business operations.
Innovation Solution
Implement asynchronous persistent inflight tracking and replay of I/O operations between primary and secondary storage clusters, allowing new I/O operations on the primary cluster without waiting for completion at the secondary cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous data replication is used between primary and secondary storage clusters, then data consistency is improved, but I/O operation delay increases during connectivity failures
Solution Approach 1:
The system performs preliminary actions by maintaining operation logs and tracking inflight I/O operations before connectivity failures occur. When connectivity is lost, the primary cluster can replay these pre-tracked operations to the secondary cluster without waiting for acknowledgment, allowing I/O operations to resume immediately while data consistency is restored through the replay mechanism.
Solution Approach 2:
The patent introduces an intermediary mechanism (operation log and inflight tracking system) that decouples the primary and secondary clusters during connectivity failures. This intermediary allows the primary cluster to continue processing I/O operations locally while maintaining a record of operations to be replayed to the secondary cluster once connectivity is restored, thus eliminating the blocking effect of synchronous replication.
2Reliability
If I/O operations are blocked during resynchronization to ensure data consistency, then data integrity is improved, but productivity decreases
Solution Approach 1:
The system performs preliminary tracking of inflight I/O operations and maintains operation logs before resynchronization is needed. This preliminary action allows the system to quickly replay operations during resynchronization without blocking new I/O operations, thus maintaining productivity while ensuring data integrity through the replay mechanism.
Solution Approach 2:
The patent enables continuity of useful action by allowing the primary cluster to continue processing new I/O operations without interruption during resynchronization. The inflight tracking mechanism ensures that operations are replayed to the secondary cluster in the correct sequence, maintaining data integrity while preventing productivity loss from blocking I/O operations.
3Speed
If asynchronous inflight replay is implemented, then I/O resumption speed is improved, but system complexity increases
Solution Approach 1:
The patent applies universality by designing the inflight tracking and replay mechanism to handle multiple scenarios: normal operation logging, connectivity failure recovery, and resynchronization. This multi-functional approach allows the same infrastructure to serve both performance improvement (fast I/O resumption) and data consistency requirements, reducing the need for separate complex systems.
Solution Approach 2:
The system uses copying by maintaining operation logs that record I/O operations to be replayed. This copy of the operation sequence allows asynchronous replay to the secondary cluster without blocking the primary cluster, achieving fast I/O resumption while managing complexity through the use of simple log copying and replay mechanisms.
Data Source
AI summary
In one embodiment, a method comprises maintaining state information regarding a data replication status for a storage object of the storage node of a primary storage cluster with the storage object being replicated to a replicated storage object of a secondary storage cluster, temporarily disallowing input/output (I/O) operations when the storage object has a connection loss or failure. The method further includes initiating a resynchronization between the storage object and the replicated storage object including initiating asynchronous persistent inflight tracking and replay of any missing I/O operations that are missing from one of a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, and allowing new I/O operations to be handled with the storage object of the primary storage cluster without waiting for completion of the asynchronous persistent inflight tracking and replay at the secondary storage cluster.