Backup System Ancestry Map for Compaction-Based Database Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Incremental backups of compaction-based databases are inefficient due to the duplication of data entries in Sorted String Tables (SSTables), as new SSTables created through compaction include data already backed up in previous SSTables, leading to wasteful storage usage.
Innovation Solution
The method involves maintaining an ancestry map to identify and refrain from backing up SSTables that are compactions of previously backed-up files, ensuring that only new data entries are copied, and using this map to recreate the SSTables as needed for restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If incremental backup copies all new SSTables regardless of compaction origin, then backup simplicity is maintained, but storage space is wasted due to duplicate data entries
Solution Approach 1:
The backup system performs preliminary analysis of each new SSTable to determine whether it is a compaction of previously backed-up files before copying. By checking ancestry information and comparing data entry timestamps in advance, the system avoids copying redundant data, thus saving storage space while maintaining backup simplicity through automated detection and decision-making.
2Quantity of substance
If compaction combines multiple SSTables into one, then storage efficiency is improved, but backup efficiency deteriorates due to potential data duplication
Solution Approach 1:
The backup system uses feedback from ancestry information and timestamp comparison to determine whether a compacted SSTable contains already-backed-up data. By analyzing the maximum timestamp of data entries in the new SSTable against previously backed-up files, the system receives feedback on redundancy and adjusts its copying decision accordingly, maintaining backup efficiency while preserving storage efficiency benefits of compaction.
3Reliability
If all new SSTables are copied in incremental backup, then data completeness is ensured, but storage redundancy increases
Solution Approach 1:
The backup system extracts and analyzes ancestry information and timestamp data from new SSTables to identify which compacted files contain data already present in previously backed-up SSTables. By taking out only the redundant data entries for exclusion from copying, the system ensures that only truly new or modified data is backed up, maintaining data completeness while eliminating storage redundancy.
Data Source
AI summary
Embodiments disclosed herein provide systems, methods, and computer readable media for sub-cluster recovery in a data storage environment having a plurality of storage nodes. In a particular embodiment, the method provides scanning data items in the plurality of nodes. While scanning, the method further provides indexing the data items into an index of a plurality of partition groups. Each partition group includes data items owned by a particular one of the plurality of storage nodes. The method then provides storing the index.