Disaster Recovery System for Linearizable Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional disaster recovery techniques for distributed data stores face challenges in maintaining low data loss windows and scalability, particularly for linearizable systems, as they often result in significant data loss and performance costs due to issues with asynchronous log shipping and synchronous geo-replication.
Innovation Solution
The implementation of a disaster recovery system (SysDR) that employs asynchronous log replication, globally synchronized clocks, batching, and a watermark service to ensure consistency and scalability, allowing for efficient asynchronous backup with near-zero lag and maintaining data store linearizability by using timestamps to preserve the correct order of updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If asynchronous log shipping is used for disaster recovery, then data loss is reduced, but ordering consistency cannot be enforced for linearizable data stores
Solution Approach 1:
The system performs preliminary actions by capturing timestamps at the time of write operations and storing them in the write-ahead log before replication. This ensures that the temporal ordering information is preserved and can be used later to enforce linearizability constraints during disaster recovery, even though the actual data replication occurs asynchronously.
Solution Approach 2:
The system implements feedback mechanisms where the backup site receives timestamps from the primary site and uses them to verify and enforce ordering constraints. The feedback loop ensures that asynchronous replication maintains consistency by continuously checking that operations are applied in the correct temporal sequence based on the captured timestamps.
2Reliability
If synchronous geo-replication is used for disaster recovery, then data consistency is maintained, but performance cost increases significantly
Solution Approach 1:
The system performs preliminary actions by capturing timestamps at the time of write operations and storing them in the write-ahead log before replication. This ensures that the temporal ordering information is preserved and can be used later to enforce linearizability constraints during disaster recovery, even though the actual data replication occurs asynchronously.
Solution Approach 2:
The system dynamically adjusts the replication strategy based on the operational context. During normal operation, asynchronous replication is used to maximize performance. When disaster recovery scenarios are detected or timestamps indicate ordering constraints, the system dynamically switches to enforcing consistency through the watermark service, achieving both performance and reliability as needed.
3Ease of manufacture
If snapshot shipping is used for disaster recovery, then implementation is simple, but data loss window becomes significant (minutes to days)
Solution Approach 1:
The system implements continuous useful action by maintaining a continuous write-ahead log that captures all write operations with timestamps in real-time. Instead of periodic snapshots, the system continuously records operational state changes, enabling near-real-time recovery with minimal data loss window while keeping the implementation relatively simple through the use of standard logging mechanisms.
Solution Approach 2:
The system performs preliminary actions by capturing timestamps at the time of write operations and storing them in the write-ahead log before replication. This ensures that the temporal ordering information is preserved and can be used later to enforce linearizability constraints during disaster recovery, even though the actual data replication occurs asynchronously.
4Productivity
If asynchronous log replication is used, then performance is maintained, but backup site state consistency becomes difficult to manage at scale
Solution Approach 1:
The system introduces a watermark service as an intermediary component that mediates between the primary site's write-ahead log and the backup site's state management. The watermark service receives timestamps from the primary site, processes ordering constraints, and coordinates the application of operations at the backup site, thereby simplifying consistency management at scale without sacrificing performance.
Solution Approach 2:
The system segments the disaster recovery process into distinct phases: timestamp capture at write time, asynchronous replication of operations, watermark-based ordering coordination, and final state application at the backup site. This segmentation allows each component to focus on specific tasks, making the overall system more manageable and scalable while maintaining performance.
Data Source
AI summary
Systems and methods are disclosed to improve disaster recovery by implementing a scalable low-loss disaster recovery for a data store. The disaster recovery system enables disaster recovery for a linearizable (e.g., externally consistent) distributed data store. The disaster recovery system also provides for a small lag on the backup site relative to the primary site, thereby reducing the data loss by providing a smaller data loss window compared to traditional disaster recovery techniques. The disaster recovery system implements a timestamp for log records based on a globally synchronized clock. The disaster recovery system also implements a watermark service that updates a global watermark timestamp that a backup node uses to apply log records.