Point-in-Time Backup Coordination Across Mutable and Immutable Datastores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing point-in-time recovery (PITR) techniques struggle to efficiently create backups of application data stored on disparate storage systems, particularly when one system uses mutable and one uses immutable data formats, leading to high costs and prolonged downtime due to service-provider implemented snapshot-like behaviors.
Innovation Solution
A system that initiates a snapshot of mutable data on one datastore, suspends delete operations on immutable data to preserve it, and performs a background copy while allowing write operations to resume, ensuring a consistent point-in-time backup is created across both datastores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If service-provider implemented snapshot-like behaviors are used for immutable data, then backup consistency is improved, but costs increase and downtime is prolonged
Solution Approach 1:
The backup process is segmented into distinct phases: a coordination phase where a coordination data structure is created to track the state of immutable data, and a background copy phase where data is copied without service interruption. This segmentation allows the critical consistency operations to be completed quickly while the bulk data transfer occurs in the background, reducing overall downtime.
Solution Approach 2:
The system performs preliminary actions by creating a coordination data structure that captures the state of immutable data before the backup process begins. This preliminary state capture allows the background copy to proceed without needing to continuously verify data consistency, enabling faster backup completion with minimal service interruption.
2Reliability
If service-provider implemented snapshot-like behaviors are used for immutable data, then backup consistency is improved, but costs increase
Solution Approach 1:
Instead of using expensive service-provider snapshot mechanisms, the system creates a coordination data structure that records the state of immutable data and performs background copying. This approach uses standard copy operations rather than proprietary snapshot features, significantly reducing costs while maintaining backup consistency through the coordination mechanism.
Solution Approach 2:
The system performs its own backup coordination and state tracking using a self-managed coordination data structure, rather than relying on service-provider snapshot features. This self-service approach eliminates the need to pay for expensive managed snapshot capabilities while achieving the same consistency goals through custom implementation.
3Reliability
If delete operations are suspended during backup of immutable data, then data consistency is improved, but productivity decreases
Solution Approach 1:
The backup operation is segmented into a coordination phase that briefly suspends delete operations to capture the state in a coordination data structure, followed by a background copy phase where delete operations resume. This segmentation limits the impact on productivity to only the brief coordination phase while maintaining consistency.
Solution Approach 2:
The system performs the necessary consistency-capturing actions (creating the coordination data structure) as a preliminary step before resuming normal operations. By completing the critical consistency operations first, the system can then allow delete operations to resume without compromising the backup integrity, minimizing productivity impact.
Data Source
AI summary
Described are techniques for creating a point-in-time backup of application data stored on disparate storage systems. The techniques include initiating a snapshot of the first datastore. The techniques further include initiating a suspension of delete operations at the second datastore to preserve immutable application data stored on the second datastore. The techniques further include initiating a background copy of the second datastore. The techniques further include initiating a resumption of the delete operations at the second datastore in response to completion of the background copy of the second datastore.