Distributed Database Backup Using Timestamp Hold and Restore Cutoff
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed databases lack effective backup capabilities across multiple clusters and nodes, leading to inconsistent data restores due to issuing new timestamps during backup procedures, resulting in incorrect data ordering and inconsistencies.
Innovation Solution
A system and method that pauses the cleanup process and timestamp issuance, generates a backup timestamp, and creates duplicate data copies, ensuring consistent backups by deleting entries older than the backup timestamp during restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If new timestamps are issued during backup procedures, then database operations continue without interruption, but data consistency is compromised leading to incorrect data ordering
Solution Approach 1:
The system performs preliminary actions by pausing the timestamp issuance process before the backup procedure begins. This ensures that no new timestamps are generated during the backup, preventing data consistency issues while maintaining operational continuity after the backup completes.
Solution Approach 2:
The system introduces an intermediary mechanism (hold on timestamp issuance) that mediates between the backup process and the timestamp generation process. This hold acts as a controlled barrier that prevents timestamp generation during backup while allowing it to resume afterward, resolving the contradiction between continuity and consistency.
2Loss of substance
If cleanup process runs during backup, then storage space is optimized by removing old entries, but backup consistency is compromised due to changing data state
Solution Approach 1:
The system performs preliminary action by pausing the cleanup process before the backup procedure starts. This ensures that the data state remains static during backup, guaranteeing consistency. After the backup completes, the cleanup process resumes to optimize storage space, thus resolving the contradiction between storage optimization and backup consistency.
Solution Approach 2:
The system introduces a hold mechanism as an intermediary between the backup process and the cleanup process. This hold prevents cleanup operations during backup while allowing them to proceed before and after, ensuring both storage optimization and backup consistency are achieved.
3Reliability
If complete data copies are stored for every backup, then restoration accuracy is ensured, but storage requirements increase significantly
Solution Approach 1:
The system segments the backup data into incremental changes rather than storing complete copies. By identifying and storing only the portions of data that changed since the last backup, the system maintains restoration accuracy while significantly reducing storage requirements.
Solution Approach 2:
The system changes the parameter of backup storage from complete data copies to incremental data changes. This parameter change allows the system to maintain full restoration capability while using significantly less storage space by only storing what has changed.
Data Source
AI summary
A distributed, transactional database uses timestamps, such as logical clock values, for entry versioning and transaction management in the database. To write to the database, a service requests a timestamp to be inserted into the database with a new version of data. During a backup procedure, a cleanup process is paused, issuing new timestamps is paused, and a backup timestamp is generated, which results in an effective backup copy. During a restore of a backup, a snapshot of the database is loaded and any entries older than the backup timestamp are deleted, which ensures that a consistent restore has occurred.