Separated Database Backup With Barriered LSM Tree Flushes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing backup process for separated databases is slow due to the need to copy entire files of LSM trees, leading to high I/O amplification and latency.
Innovation Solution
A method involving the creation of a new backup LSM tree associated with a new point in time, sharing a mutable key-value database, which includes a barriered flush mechanism to reduce I/O amplification and latency by selectively flushing only changed key-value pairs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire LSM tree files are copied during backup, then backup completeness is ensured, but backup speed decreases and I/O amplification increases
Solution Approach 1:
The patent segments the backup process by separating keys from values in the LSM tree. Instead of copying entire LSM tree files, the system copies only the key metadata and then retrieves only the necessary value data on demand. This segmentation enables selective copying of data, dramatically reducing the amount of data transferred during backup while maintaining completeness through the separated key-value architecture.
Solution Approach 2:
The patent extracts the key component from the LSM tree file structure and separates it from the values. By taking out the key metadata and storing it separately, the system can perform backup operations on just the key structure rather than the entire file. This extraction allows the backup process to focus only on essential metadata while values are fetched selectively, reducing I/O amplification and improving backup speed.
2Reliability
If entire LSM tree files are copied during backup, then data integrity is maintained, but I/O amplification increases
Solution Approach 1:
The system segments the data into separate key and value components, allowing backup operations to target only the key metadata structure. This segmentation means that when backing up, the system transfers only the compact key information rather than the full LSM tree files, significantly reducing the volume of data moving through the I/O system and thereby reducing I/O amplification while preserving data integrity through the separated architecture.
Solution Approach 2:
The patent extracts and separates the key metadata from the value data, enabling the backup process to operate on the much smaller key structure. By taking out the key component and storing it independently, the system minimizes the data that needs to be copied during backup, directly reducing I/O amplification and energy consumption while maintaining data integrity through the preserved key-value relationships.
3Loss of energy
If barriered flush mechanism is implemented, then I/O amplification is reduced, but system complexity increases
Solution Approach 1:
The barriered flush mechanism performs preliminary actions by establishing a barrier state before flushing data. This barrier ensures that all necessary data is prepared and locked in memory before the actual flush operation begins. By performing this preliminary barrier setup, the system can then execute the flush operation more efficiently without needing to repeatedly check or re-flush data, reducing overall I/O amplification despite the added complexity of the barrier management mechanism.
Data Source
AI summary
A method for performing a backup in relation to a separated database, the method includes creating, by a processing circuit, a new backup log structured merge (LSM) tree of the separated database, wherein the new backup LSM tree is associated with a new point in time (PIT) and belongs to a group of LSM trees that comprise a primary LSM tree and one or more backup LSM trees that are associated with different PITs, and share a key value (KV) database that is mutable and is separated from the group; wherein the creating comprises storing the new backup LSM tree to a destination.