Distributed Transactional Database Backup With Timestamped Restore
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed databases lack robust backup capabilities across multiple clusters and nodes, leading to inconsistent data restores due to issues like out-of-order writes during backup procedures, resulting in data inconsistencies.
Innovation Solution
A system and method that pauses the cleanup process and timestamp issuance during backups, generates a backup timestamp, and creates duplicate data copies, ensuring consistent backups by deleting entries older than the backup timestamp during restore.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the cleanup process continues during backup, then productivity is maintained, but data consistency deteriorates due to out-of-order writes
Solution Approach 1:
The system performs preliminary actions by pausing the cleanup process and timestamp issuance before the backup begins. This ensures that no new timestamps are generated and no cleanup operations occur during the backup, preventing out-of-order writes and ensuring data consistency. The backup timestamp is captured at this point to mark the consistent state.
Solution Approach 2:
The backup timestamp acts as an intermediary mechanism that mediates between the cleanup process and the backup operation. By comparing entry timestamps against the backup timestamp, the system determines which entries should be included in the backup, ensuring consistency without requiring complete system shutdown.
2Reliability
If timestamp issuance is held during backup, then data consistency is improved, but productivity decreases due to service interruption
Solution Approach 1:
The system performs preliminary actions by pausing timestamp issuance only for the brief duration needed to capture the backup timestamp, not for the entire backup process. This minimizes service interruption while ensuring that no new timestamps are generated during the critical backup window, maintaining data consistency.
Solution Approach 2:
The timestamp issuance is paused periodically only during the brief window when the backup timestamp is being captured, rather than being paused continuously. This periodic, brief interruption maintains backup consistency while minimizing impact on service availability and productivity.
3Reliability
If all data is copied during backup, then reliability is improved, but productivity decreases due to full data duplication
Solution Approach 1:
Instead of uniformly copying all data, the system applies local quality by selectively copying only those entries whose timestamps are earlier than the backup timestamp. This ensures backup completeness for relevant data while improving backup speed by avoiding duplication of unnecessary entries.
Solution Approach 2:
The system performs partial action by copying only the necessary portion of data (entries with timestamps before the backup timestamp) rather than all data. This partial backup approach maintains reliability for the required data scope while significantly improving backup productivity.
Data Source
AI summary
A distributed, transactional database uses timestamps, such as logical clock values, for entry versioning and transaction management in the database. To write to the database, a service requests a timestamp to be inserted into the database with a new version of data. During a backup procedure, a cleanup process is paused, issuing new timestamps is paused, and a backup timestamp is generated, which results in an effective backup copy. During a restore of a backup, a snapshot of the database is loaded and any entries older than the backup timestamp are deleted, which ensures that a consistent restore has occurred.