Active/Active Cluster Replication Lock Arbitration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Active/active replication in storage systems faces challenges with deadlocks during concurrent updates and maintaining data consistency and safe IO recovery when components fail.
Innovation Solution
Designate one cluster as a lock winner, prioritize it for locking, and implement a replication journal for atomic data persistence and recovery, ensuring consistent IO operations across clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If active/active replication is implemented to enable concurrent IO operations across clusters, then continuous availability and productivity are improved, but deadlock issues and data consistency problems occur when both clusters attempt to lock the same page simultaneously
Solution Approach 1:
The patent designates one cluster as the 'lock winner' that has priority rights to acquire locks on pages, while the other cluster becomes the 'lock loser'. This asymmetric designation resolves deadlocks by ensuring that when both clusters simultaneously attempt to lock the same page, only the lock winner can successfully acquire the lock, preventing the circular wait condition that causes deadlocks. The lock winner designation is stored in cluster configuration and used by the lock management module to make arbitration decisions.
Solution Approach 2:
The patent introduces a replication journal as an intermediary mechanism to ensure data consistency. The lock winner cluster creates journal entries atomically before applying write operations, and the lock loser cluster replays these journal entries to maintain data consistency. This intermediary journaling system allows the system to recover safely from failures and maintain consistency even when clusters fail at different points during the locking and writing process.
2Reliability
If locking mechanisms are implemented to prevent deadlocks, then data consistency is improved, but system complexity and difficulty of operation increase due to the need for lock management and failure recovery protocols
Solution Approach 1:
The patent implements local journaling at each cluster, where the lock winner cluster maintains a replication journal locally. This local quality approach allows each cluster to independently manage its own journal entries and recovery process without requiring complex centralized coordination. The lock management module at each cluster uses the local lock winner designation to make decisions, simplifying the overall system architecture while maintaining data consistency.
3Productivity
If both clusters attempt to acquire locks simultaneously for concurrent updates, then productivity and continuous availability are improved, but deadlock occurs leading to loss of time for resolving conflicts
Solution Approach 1:
The system performs preliminary action by designating the lock winner cluster before any IO operations occur. This pre-established hierarchy allows the system to immediately determine lock acquisition rights without requiring complex runtime arbitration when deadlocks are detected. The lock management module can quickly identify the lock winner based on the pre-stored designation, minimizing the time lost to deadlock resolution while still allowing both clusters to attempt concurrent updates.
Data Source
AI summary
In one aspect, IO recovery mechanisms in active/active replication for storage clusters provide, for each write IO request: determining an extent of pages to be modified; acquiring a lock by a cluster local to the request and locally persisting data; atomically creating an entry in a local journal; and sending, by the local cluster, a data write request to a peer cluster. Upon receiving acknowledgement from the peer cluster of successful transmission, the replication journal entry is cleared, the extent is unlocked, and a notification is returned to the peer. In response to a failure event at the peer cluster after persisting the data at the local cluster and creating the journal entry at the local cluster, an aspect includes sending, to the peer cluster upon recovery of the peer cluster, the journal entry with respect to an extent corresponding to a time of the failure event.