Edge Key-Value Store Deduplication via Bloom Filter Compaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log-structured merge tree (LSM tree)-series key-value stores face a space amplification problem in update-intensive workloads due to delayed deletion and increased deprecated data, leading to inefficient use of storage space.
Innovation Solution
A data deduplication-extended compaction method is introduced for key-value stores, utilizing a Bloom filter to check for duplicated data during compaction, which removes duplicates when the deduplication ratio reaches a threshold, thereby reducing space amplification and maintaining a balance with write amplification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If delayed deletion is implemented in LSM tree, then write amplification is reduced, but space amplification increases
Solution Approach 1:
The patent segments the compaction process into two distinct phases: a first compaction process that handles metadata compaction and SSTable relocation, and a second compaction process that performs deduplication. This segmentation allows the system to achieve both reduced write amplification (through delayed deletion in the first phase) and reduced space amplification (through deduplication in the second phase) without compromising either objective.
2Volume of stationary object
If deduplication is performed during compaction, then space amplification is reduced, but I/O costs increase
Solution Approach 1:
The patent performs preliminary actions by first completing the first compaction process (metadata compaction and SSTable relocation) before initiating the second compaction process for deduplication. This preliminary completion of essential compaction tasks ensures that the system structure is stabilized, allowing the deduplication process to operate more efficiently with reduced I/O overhead, as the basic compaction framework is already in place.
3Reliability
If full compaction is performed, then data consistency is improved, but write amplification increases
Solution Approach 1:
The patent applies partial action by implementing a two-stage compaction process where the first stage performs essential compaction operations (metadata compaction and SSTable relocation) to achieve sufficient data consistency, while the second stage performs optional deduplication. This partial completion of compaction in the first stage maintains data consistency requirements while avoiding the excessive action of performing full compaction including deduplication in every write operation, thus reducing write amplification.
Data Source
AI summary
Disclosed is a data deduplication method for an edge computer. The method is performed in a key-value store, and may include receiving a compaction request occurred from the key-value store to a metadata layer, checking whether deduplication for removing duplicated data is required when compaction of a metadata file is performed in response to the received compaction request, and removing the duplicated data by checking whether the deduplication is required.