Oplog Index Replication for Cluster Failover Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face latency issues and restricted log sizes due to time-consuming recovery of log indices upon node failure, limiting data replication and I/O access efficiency in clustered storage environments.
Innovation Solution
A technique for replicating the operations log (oplog) index across nodes in a cluster, using a binary search tree for fast failover and batch processing of metadata, ensuring continuous synchronization and recovery without downtime, by employing a replication factor algorithm for global redundancy and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log index is rebuilt upon node failure, then data access is restored, but recovery time increases and I/O is interrupted
Solution Approach 1:
The patent pre-replicates the log index from the primary node to secondary nodes before failure occurs. When a node fails, the secondary node already possesses a current copy of the log index, eliminating the need for time-consuming reconstruction and enabling immediate data access restoration.
Solution Approach 2:
The patent creates and maintains copies of the log index on secondary nodes through replication mechanisms. These copies serve as ready-to-use alternatives when the primary node fails, avoiding the need to rebuild the index from scratch and significantly reducing recovery time.
2Quantity of substance
If log size is increased to store more data, then data retention is improved, but recovery time increases excessively
Solution Approach 1:
By replicating the log index to secondary nodes, the system can maintain larger log sizes for improved data retention while keeping recovery time manageable. The copied indexes on secondary nodes enable fast failover regardless of the primary log's size, as the secondary node can immediately take over with its pre-cached index.
Solution Approach 2:
The system performs preliminary replication of log index data to secondary nodes, allowing the primary node to accumulate larger amounts of log data without proportionally increasing recovery time. The pre-positioned copies on secondary nodes ensure that recovery speed remains independent of log size.
3Loss of time
If log index is replicated to secondary node, then failover time is reduced, but system complexity increases
Solution Approach 1:
The patent implements log index replication to secondary nodes, creating simplified copies of the index structure. This approach reduces failover time by having ready-to-use indexes available, while the copying mechanism itself provides a relatively simple implementation compared to full synchronization approaches.
4Reliability
If acknowledgement is sent after persistent storage, then data reliability is ensured, but I/O latency increases
Solution Approach 1:
The patent uses a log structure to record data before writing to the backend storage tier. The log index tracks these preliminary recordings, allowing the system to acknowledge I/O operations once data is in the log (faster) while maintaining the ability to restore data reliability through the replicated index and log recovery mechanisms.
Data Source
AI summary
A technique replicates an index of an operations log (oplog) from a primary node to a secondary node of a cluster in the event of failure. The oplog functions as a staging area to coalesce random write operations directed to a virtual disk (vdisk) stored on a backend storage tier. The oplog temporarily caches write data as well as metadata describing the write data. The metadata includes descriptors to the write data corresponding to offset ranges of the vdisk and are used to identify ranges of write data for the vdisk that are cached in the oplog. To facilitate fast lookup operations of whether write data is cached in the oplog, an oplog index provides a state of the latest data for offset ranges of the vdisk that enables fast failover of metadata used to construct the oplog index in memory without downtime or significant metadata replay.