LSM Tree Data Deduplication for Backup Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data deduplication methods during backup processes are resource-intensive and time-consuming due to the overhead associated with searching for and adding new data blocks, particularly when using conventional indexing techniques that require frequent disk access.
Innovation Solution
The implementation of at least two log-structured merge (LSM) trees, where one tree is stored in volatile memory for quick searches and the other maps segment IDs to hash values, reducing the size of records to fit entirely in memory and minimizing disk operations, allowing for efficient addition and search of data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional indexing techniques are used for data deduplication, then data blocks can be tracked and deduplicated, but the search and insert operations become resource-intensive and time-consuming due to frequent disk access
Solution Approach 1:
The patent divides the indexing structure into two separate LSM trees: one optimized for search operations (storing reduced hash values) and one optimized for insert operations (storing full hash values and segment IDs). This segmentation allows each tree to be specialized for its specific function, reducing the overhead of general-purpose indexing and enabling parallel optimization of search and insert paths.
Solution Approach 2:
The patent introduces a dimensional transformation by creating reduced hash values (a compressed representation) from full hash values. This reduced dimension is used specifically for search operations in the first LSM tree, while the full-dimensional data remains in the second LSM tree. This dimensional change enables faster search operations with smaller data footprint in volatile memory.
2Speed
If data is stored in volatile memory for fast access, then search operations become faster, but the amount of memory required increases and data must be managed carefully
Solution Approach 1:
The patent extracts only the essential search key (reduced hash value) from the full data structure and places it in the first LSM tree in volatile memory. The complete data (full hash value, segment ID, physical offset) remains in the second LSM tree on disk. This extraction allows the volatile memory structure to be compact and fast, while the full data resides on disk where it belongs.
Solution Approach 2:
The reduced hash value acts as an intermediary between the volatile memory search structure and the disk-based full data structure. It enables fast comparison and matching in memory without requiring the full data structure to be loaded into memory, thus mediating between the speed requirements of volatile memory and the completeness requirements of disk storage.
3Reliability
If frequent disk operations are performed for backup operations, then data can be persisted, but the backup process becomes slower and more resource-intensive
Solution Approach 1:
The patent performs preliminary actions by maintaining two LSM trees in advance: one with reduced hash values ready for immediate search comparison, and another with full data ready for immediate insert operations. When a deduplication operation is needed, the system can immediately query the first tree and insert into the second without performing data transformation or disk I/O during the critical path operations.
Solution Approach 2:
The patent changes the parameter representation of hash values based on the operation type: reduced hash values (fewer bits, smaller size) are used for search operations to reduce memory bandwidth and comparison time, while full hash values are used for insert operations to ensure data integrity and uniqueness. This parameter change optimizes each operation type for its specific requirements.
Data Source
AI summary
Disclosed are systems, methods and computer program products for data deduplication during a backup using at least two LSM trees. An example method includes calculating, for a first data block, a first hash value associated with the first data block and determining a reduced hash value based on the first hash value. The method includes determining whether the first data block contains data duplicative of an existing data block in a prior backup based on whether the reduced hash value occurs in a first log-structured merge (LSM) tree. If so, the method includes comparing the first hash value to one or more hash values in a second LSM tree to identify a matching hash value, and writing a first segment identifier (ID) corresponding to the matching hash value in an archive, the first segment ID referencing the existing data block in a segment store.