KV-Store SSTable Compaction via Delete Log Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current key-value store (KV-Store) systems face inefficiencies in compaction operations, requiring significant I/O bandwidth and memory resources, which hinders other operations and affects user experience due to the need to read and process multiple SSTables simultaneously.
Innovation Solution
Implementing a method for single-file compaction by using a Delete Log to identify and remove non-latest values from SSTables, reducing the need to read multiple SSTables during compaction and thereby minimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If multi-file compaction is performed by merging multiple SSTables simultaneously, then storage space is reduced by removing non-latest values, but I/O bandwidth and memory resources are heavily occupied affecting other operations
Solution Approach 1:
The patent segments the compaction process by introducing per-SSTable Delete Logs that independently track deleted keys for each SSTable. This allows the system to process and compact SSTables individually rather than requiring simultaneous processing of multiple SSTables, thereby reducing the peak I/O bandwidth and memory resource requirements while maintaining the space-saving benefits of compaction.
2Reliability
If multiple SSTables are read simultaneously during compaction, then non-latest values are identified and removed, but resource consumption increases and user experience deteriorates
Solution Approach 1:
The patent applies preliminary action by maintaining Delete Logs that continuously record deleted keys as they occur during PUT operations. This pre-computation of deletion information eliminates the need to read and analyze multiple SSTables simultaneously during compaction, as the Delete Logs already contain the necessary information to identify non-latest values, thereby reducing resource consumption while ensuring data consistency.
Data Source
AI summary
Example methods and apparatuses for file compaction in a key-value store system relating to the field of data processing technologies are described. One example method for file compaction in a key-value store (KV-Store) system includes compacting, according to a to-be-deleted log (Delete Log) that corresponds to a to-be-compacted sorted string table (SSTable), the to-be-compacted SSTable to generate a new SSTable. A key corresponding to a non-latest value in the KV-Store system and stored in the to-be-compacted SSTable is recorded in the Delete Log, and the new SSTable does not include a key-value pair that corresponds to the key in Delete Log. The to-be-compacted SSTable is then deleted.