Master-less Database Cluster Backup Without Quiescing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficient backup and restore of distributed, master-less NoSQL database clusters is challenging due to the need for cluster consistency without quiescing database applications, managing inconsistencies across multiple nodes, and optimizing storage space usage.
Innovation Solution
A two-phase backup process that transforms crash-consistent snapshots into cluster-consistent backups without quiescing the database, using token information to reconcile commit logs and data files, and performing compaction to remove duplicates, allowing for efficient storage and restore operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional backup methods are used on distributed database clusters, then backup operations can be performed, but the database applications must be quiesced (stopped) to ensure consistency
Solution Approach 1:
The backup process is divided into multiple phases: first capturing crash-consistent snapshots from individual nodes, then reconciling commit logs and data files across nodes to achieve cluster consistency. This segmentation allows backup operations to proceed without requiring the entire database system to be stopped.
Solution Approach 2:
The system performs preliminary actions by capturing snapshots and commit logs before reconciliation occurs. Token information is collected and stored in advance, enabling the subsequent compaction phase to efficiently reconcile data across nodes without interrupting database operations.
2Reliability
If snapshots are taken from multiple database nodes, then backup coverage is improved, but inconsistencies arise across different nodes that require resource-intensive repairs
Solution Approach 1:
Token information serves as an intermediary that tracks which data files belong to which nodes. This token metadata enables the compaction process to efficiently reconcile snapshots across nodes by identifying duplicates and inconsistencies without requiring intensive computational repairs, as the tokens provide a roadmap for reconciliation.
Solution Approach 2:
The system changes the state of data files from crash-consistent to cluster-consistent by flushing commit logs and performing compaction. This parameter change transforms the backup from an incomplete state to a consistent state, eliminating the need for resource-intensive post-processing repairs.
3Reliability
If data from multiple nodes is stored separately, then node-level backup is achieved, but storage space efficiency decreases due to duplicate data
Solution Approach 1:
The compaction process merges data from multiple node snapshots into a unified backup structure. By combining data and eliminating duplicates through the token-based reconciliation system, the system maintains complete backup integrity while significantly reducing the total storage space required compared to storing separate node backups.
Data Source
AI summary
Methods and systems for generating a cluster consistent backup of a distributed, master-less database cluster is provided. One method includes a first phase of a backup operation where a crash consistent backup of the various LUNs used by the cluster nodes is taken. The crash consistent backup is then transformed into a cluster consistent backup during a second phase of the backup operation.